2 min lesson
Prefix caching: reuse the work you already did
Compare both sides of "Prefix caching: reuse the work you already did", then name the condition that decides between them.
Step 1 of 2
Prefix caching: reuse the work you already didthe lever with Cursor's name on it
If two requests start with the same tokens, their KV cache for that shared span is identical. Prefix caching keeps those computed keys/values around and reuses them, so a request that shares a prefix skips prefilling it entirely. The first matching token is then the only new prefill work.
Cursor's prompts are unusually repetitive, which is exactly the condition prefix caching loves. The same system prompt, the same tool definitions and often the same file or repo context get sent across many requests in a session. A high prefix-cache hit rate turns an expensive full prefill into a near-free cache read.
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
The same request, two cache outcomes - and why affinity routing decides which one you get.
Learn more
Full explanation
Full explanation
Connect prefix caching to routing explicitly: prefix-cache affinity means "send requests that share a prefix to the same instance." If you load-balance round-robin, every instance misses on the shared prefix and you pay full prefill everywhere. Affinity-aware routing - hashing on the prefix or session - is a real lever a routing engineer owns and naming it shows you connect mechanics to the system you'd build.
Prefix caching only helps for an exact token-prefix match from position zero. Change one token early in the prompt - reorder the system message, inject a timestamp, vary the tool list - and the cache misses from that point on. "Keep the stable, shared content at the front and the volatile content at the back" is the correctness rule that makes the cache actually pay off.
QWhy does prefix-cache affinity push you away from naive round-robin load balancing?