2 min lesson
Latency, cost and inference at product scale
Rebuild the sequence in "Latency, cost and inference at product scale" from memory, ending with the check that proves the outcome.
Step 1 of 3
Tab prediction has to land before the user's next keystroke. That single constraint - sub-100ms, every keystroke, millions of users - is one of the most common design prompts for this role and it rewards engineers who think in tail latency and unit economics.
“Design Cursor's tab prediction system: sub-100ms latency, on every keystroke, across millions of users.”
Don't jump to a model. Start by sizing the problem out loud, because the numbers dictate the architecture. Every keystroke at millions of users is a torrent of requests, most of which the user will never see - so the cheapest request is the one you never make.