1 min lesson
A worked design: the request's journey
Put this idea into your own words: "The strongest onsite design answer is a single request narrated end to end."
Step 1 of 3
The strongest onsite design answer is a single request narrated end to end. Trace one completion from the keystroke to the streamed token and show where every technique from this module plugs in.
This is the backbone of the systems-design round. Walk the happy path once, then walk the failure branches, naming the mechanism at each step rather than gesturing at "the system handles it."
Learn more
Full explanation
One Request, Keystroke to Token
Interactive diagram. Step through it with the Next and Previous controls below, or Tab to a region to read its detail.
Each stage owns exactly one class of failure - that's how the pieces compose without overlap.
Learn more
Advanced table
Reference table
Where each technique plugs in
- Stage
- Admission
- Technique
- Rate limit + load shed
- Failure it prevents
- Spike cascading into the provider
- Stage
- Routing
- Technique
- Prefix-cache affinity + health routing
- Failure it prevents
- Wasted recompute; routing to a sick upstream
- Stage
- Gateway
- Technique
- Timeout + hedge at p95
- Failure it prevents
- A slow provider blowing the latency budget
- Stage
- Provider call
- Technique
- Circuit breaker
- Failure it prevents
- Hammering a dying upstream
- Stage
- Retry path
- Technique
- Budgeted retry → fallback ladder
- Failure it prevents
- A provider outage becoming user-visible
| Stage | Technique | Failure it prevents |
|---|---|---|
| Admission | Rate limit + load shed | Spike cascading into the provider |
| Routing | Prefix-cache affinity + health routing | Wasted recompute; routing to a sick upstream |
| Gateway | Timeout + hedge at p95 | A slow provider blowing the latency budget |
| Provider call | Circuit breaker | Hammering a dying upstream |
| Retry path | Budgeted retry → fallback ladder | A provider outage becoming user-visible |
Read it as a technique-to-failure map: each stage's guard and the exact failure it removes, with the retry path broken out as its own row.
Tab versus Agent: same path, different dials
One pipeline serves both, but the parameters invert. Contrasting them shows you understand that the design is a set of dials, not a single fixed policy.
Tiny prompt, sub-100ms budget, latency-critical.
Hedge aggressively; tight timeout; smaller fallback model is fine.
Cheap enough that a duplicate from hedging barely registers.
Long, expensive, multi-step, cancellation-prone.
Hedge rarely; fail over to an equivalent-tier model, not a weaker one.
Propagate cancellation hard so an abandoned turn stops burning tokens.