1 min lesson
Latency budgeting end to end
Walk through each part of "Latency budgeting end to end", then explain what each one does.
Step 1 of 2
Cursor is latency-obsessed by design and it shows in the interview. "Make it fast" is a junior answer. "Here's my millisecond budget and where I'd spend or save" is the one that lands.
Break the round trip into named components, give each a budget and treat the sum as a contract. When you blow the budget, you'll know exactly which segment to attack instead of guessing.
- DNS
- ~0–20 ms - usually cached on the client after first resolve
- TLS handshake
- ~0–30 ms - amortized to near-zero with connection reuse / 0-RTT
- Edge processing
- ~5–15 ms - WAF, rate-limit check, routing
- Network (user↔region)
- ~20–80 ms - dominated by physical distance; the case for regional proximity
- Queueing / scheduling
- ~5–40 ms - grows fast under saturation; the silent p99 killer
- Backend processing
- the remainder - your actual work; for inference this dwarfs everything
These are illustrative, not Cursor's real numbers - the point is to reason in named segments with explicit budgets.
Inference is its own world: a streamed completion's time-to-first-token matters more than total time, because the user sees motion immediately. Budget the first token tightly and let the stream fill in.