1 min lesson
Latency budgeting end to end
Walk through each part of "Latency budgeting end to end", then explain what each one does.
Step 1 of 2
The infra job description asks for smart tradeoffs between cost and reliability, and a millisecond budget is how you show one in a design round. "Make it fast" is a junior answer. "Here's my millisecond budget and where I'd spend or save" is the one that lands.
Break the round trip into named components, give each a budget and treat the sum as a contract. When you blow the budget, you'll know exactly which segment to attack instead of guessing.
- DNS
- ~0 to 20 ms, usually cached on the client after first resolve
- TLS handshake
- ~0 to 30 ms, amortized to near-zero with connection reuse / 0-RTT
- Edge processing
- ~5 to 15 ms for the WAF, rate-limit check and routing
- Network (user↔region)
- ~20 to 80 ms, dominated by physical distance. This is the case for regional proximity.
- Queueing / scheduling
- ~5 to 40 ms that grows fast under saturation. It's the silent p99 killer.
- Backend processing
- The remainder: your actual work. For inference this dwarfs everything.
These numbers are illustrative and are not Cursor's. The point is to reason in named segments with explicit budgets.
Inference is its own world: a streamed completion's time-to-first-token matters more than total time, because the user sees motion immediately. Budget the first token tightly and let the stream fill in.