1 min lesson
Tail latency and the hot path
Say what this means in practice: "Cursor's request path is always hot and always user-facing."
Step 1 of 4
Cursor's request path is always hot and always user-facing. Optimize for the tail, because the p99 is what a developer feels - and for Tab, the whole budget is under 100ms, so the tail is the product.
The mean lies. If p50 is 40ms and p99 is 600ms, one in a hundred completions arrives after the user has typed three more characters and they experience that as broken even though the average looks great. Talk in percentiles or an inference interviewer will assume you've never operated a serving path.
Learn more
Advanced table
Every tail source has a named fix
- Source of tail
- Cold connections
- Why it spikes p99
- TLS + TCP handshake on first request to a provider
- Mitigation
- Connection pooling + keep-alive; pre-warm pools
- Source of tail
- GC / runtime pauses
- Why it spikes p99
- A stop-the-world pause lands on an unlucky request
- Mitigation
- Limit allocations on the hot path; tune the runtime
- Source of tail
- Head-of-line blocking
- Why it spikes p99
- A slow request stalls others behind it
- Mitigation
- Per-request isolation; bounded concurrency, not one queue
- Source of tail
- Queueing delay
- Why it spikes p99
- Request waits before any work starts
- Mitigation
- Shed early; size queues to the latency budget
- Source of tail
- Slow provider
- Why it spikes p99
- One upstream's tail bleeds into yours
- Mitigation
- Hedge to a second provider at measured p95
| Source of tail | Why it spikes p99 | Mitigation |
|---|---|---|
| Cold connections | TLS + TCP handshake on first request to a provider | Connection pooling + keep-alive; pre-warm pools |
| GC / runtime pauses | A stop-the-world pause lands on an unlucky request | Limit allocations on the hot path; tune the runtime |
| Head-of-line blocking | A slow request stalls others behind it | Per-request isolation; bounded concurrency, not one queue |
| Queueing delay | Request waits before any work starts | Shed early; size queues to the latency budget |
| Slow provider | One upstream's tail bleeds into yours | Hedge to a second provider at measured p95 |
Every tail source has a named fix - reciting the source without the fix is half an answer.