CPU vs GPU Inference and Performance

Build the mental model

Running an LLM means repeatedly multiplying large matrices — the model's weights against the current activations — once per layer, for every token generated. This is exactly the kind of workload GPUs were built for.

A matrix multiply decomposes into millions of independent multiply-add operations, and a GPU has thousands of small cores that can run many of them at once, while a CPU has only a handful of cores optimized for sequential logic instead. That parallelism gap is why GPU inference is dramatically faster for the same model — not any difference in 'intelligence' or precision; the arithmetic is identical either way.

When a model does not fit entirely in VRAM, layer offloading splits it: some transformer layers run on the GPU, the rest run on the CPU, with activations shuttling across the PCIe bus between them.

This lets you run models larger than your GPU's memory, but every layer left on the CPU (and every trip across PCIe) adds latency, so offloading is a graceful-degradation tool, not a free lunch.

Two phases behave differently and are worth naming separately: prefill and decode.

The KV cache exists to make decode fast: it stores each previous token's key and value vectors so they are not recomputed every step. But that cache grows linearly with context length, consuming more VRAM as the conversation gets longer, and can eventually become the actual memory bottleneck, not the model weights themselves.

KV Cache

The stored key and value vectors from all previous tokens in a generation, kept so the model doesn't recompute them at every decoding step — the main reason decode is fast, and the main reason long contexts use a lot of memory.

TTFT (Time To First Token)

The time between sending a prompt and receiving the first generated token back — dominated by the prefill phase, which processes the entire prompt before generation can begin.

Tokens/sec

The steady-state generation speed during decode, once the first token is out — measures how fast the model produces each subsequent token, one at a time.

MODEL SPLIT ACROSS GPU/CPU + GROWING KV CACHE
---------------------------------------------
MODEL SPLIT ACROSS GPU/CPU + GROWING KV CACHE
-----------------------------------------------

  GPU (fast, limited VRAM)     CPU (slower, more RAM)
  +---------------------+     +---------------------+
  | layer 1  ... layer 24|    | layer 25 ... layer 32|
  +---------------------+     +---------------------+
            |                          |
            +------ activations -------+
                 (crosses PCIe bus)

  KV CACHE (grows as tokens are generated)
  [t1][t2][t3][t4][t5] ... -> cache size keeps growing
  more context tokens = more VRAM used by the cache alone

Connect it to a real scenario

Two settings control most of the CPU/GPU tradeoff in tools like llama.cpp, Ollama, and LM Studio: how many layers to offload to GPU, and the context length you allow.

Offload as many layers as fit in VRAM, leaving headroom for the KV cache and activations — VRAM usage is not just weights. If you hit an out-of-memory error, reduce offloaded layers before reducing context, since context affects both prefill time and KV cache size directly.

Watch TTFT and tokens/sec as separate numbers when you benchmark. A longer prompt raises TTFT (more prefill work) without necessarily changing steady-state tokens/sec. A larger KV cache from a long conversation can slow decode as memory bandwidth becomes the bottleneck, even with an unchanged model.

Quantization speeds up both phases and shrinks memory, at some cost to quality — usually a good first lever before touching layer counts or context limits.

• GPUs are faster at inference because matrix multiplication parallelizes extremely well across thousands of cores — the math is the same as on CPU, just spread wider.
• Prefill (processing the prompt, measured as TTFT) and decode (generating tokens one at a time, measured as tokens/sec) are separate phases with different bottlenecks.
• The KV cache trades memory for decode speed, and its size grows linearly with context length — long contexts cost real VRAM even before the model finishes responding.

Try the working example

def kv_cache_mb(num_layers, num_heads, head_dim, context_len, bytes_per_val=2):
    # KV cache holds one Key vector and one Value vector per token per layer.
    # bytes = layers * 2 (K and V) * heads * head_dim * context_len * bytes_per_val
    total_bytes = num_layers * 2 * num_heads * head_dim * context_len * bytes_per_val
    return total_bytes / (1024 * 1024)


# Rough shape resembling a 7B-class dense transformer (32 layers, 32 heads, 128 head_dim)
LAYERS, HEADS, HEAD_DIM = 32, 32, 128

for ctx in [512, 2048, 8192, 32768]:
    mb = kv_cache_mb(LAYERS, HEADS, HEAD_DIM, ctx)
    print(f"context={ctx:>6} tokens -> approx KV cache = {mb:.1f} MB (fp16)")

Running this against a 32-layer, 32-head, 128-head-dim shape (roughly a 7B-class dense model) at fp16 gives:

context=   512 tokens -> approx KV cache = 256.0 MB (fp16)
context=  2048 tokens -> approx KV cache = 1024.0 MB (fp16)
context=  8192 tokens -> approx KV cache = 4096.0 MB (fp16)
context= 32768 tokens -> approx KV cache = 16384.0 MB (fp16)

This is a formula-based approximation, not a measurement from a real model — actual numbers depend on the specific architecture (grouped-query attention, for example, uses far fewer KV heads than query heads and shrinks these numbers substantially) and on the inference engine's memory layout.

Hugging Face — LLM Inference Optimization — Local AI / Local LLM