1 min lesson
Decoding and the KV cache
Think through this situation: "A teammate says 'we can serve longer context for free, the weights don't change size.' What do you push back with?" Give the practical answer in plain words.
Step 1 of 2
Decoding and the KV cachewhere inference memory actually goes
Generation is autoregressive: one token at a time, each conditioned on all prior tokens through a causal mask. Recomputing every prior key and value at each step would be quadratic, so you cache them. The KV cache stores the keys and values for every past token across every layer and head.
- Causal mask
- Token t may only attend to positions <= t. During generation the new token attends to the whole cache.
- KV cache size
- Scales as 2 x layers x heads x head_dim x seq_len x batch. At long context this dwarfs the weights and caps how many concurrent users you serve.
- Sampling
- Temperature flattens or sharpens the logits; top-p keeps the smallest set of tokens covering probability p. Lower both when you want deterministic, tool-correct edits.
Ask any inference-cost question by starting with KV-cache memory, not parameter count.
A common miss is calling parameters the inference bottleneck. At long context with many concurrent sessions, the KV cache is what runs you out of GPU memory first. This is exactly why grouped-query attention and cache-eviction tricks exist and why a Cursor researcher serving long agent sessions cares about it.