2 min lesson
KV cache and prefix caching
Show in two sentences why the KV cache, rather than raw compute, usually limits how many concurrent requests a single GPU can serve.
Step 1 of 2
The KV cache is the single most important object in inference economics. It is why decode is fast, it is your tightest memory constraint and prefix caching is the biggest cost lever Cursor has.
Attention needs the keys and values of every previous token to compute the next one. Without a cache, each new token would re-run attention over the entire sequence from scratch - quadratic work that would make decode hopelessly slow. The KV cache stores those keys and values per token, so generating token N is cheap: attend to the cached state plus the one new token.
That speed has a price in memory. The cache grows linearly with context length and with how many requests share the GPU at once.
- Roughly proportional to
- context_length × batch_size × layers × hidden_size × 2 (K and V)
- Grows with
- Longer contexts and more concurrent requests
- Lives in
- GPU HBM, alongside model weights - they compete for the same space
- Why it bounds concurrency
- When KV memory fills, you can't admit another request - this caps batch size
The KV cache, not raw FLOPs, is usually what limits how many requests one GPU can serve.
Because the cache is the capacity ceiling, how you manage its memory directly sets how many concurrent users a GPU can hold. Naive allocation reserves the full context window per request up front, which wastes huge amounts of memory on requests that finish short. PagedAttention, introduced by vLLM, fixes this by chopping the cache into fixed-size pages allocated on demand, the way an OS pages physical memory. Fragmentation drops, utilization rises and effective batch size goes up without buying more GPUs.