2 min lesson
The failure branches
Imagine this comes up at work: "In a design round, why is narrating a single request through the system a stronger approach than describing each component in turn?" Start with the practical move.
Step 1 of 2
The failure branches
The happy path is the easy half. Naming the branches - and the clean exit when there's no good option left - is what closes a strong design answer.
- Provider returns 5xx or times out → classify as retryable, spend from the retry budget, cross-route to the next provider on the ladder.
- Over admission budget → shed fast with an explicitly-retryable error and a backoff hint, never queue past the latency budget.
- Oversized context → down-route to a larger-window model or reject with a clear, actionable error before wasting a provider call.
- Last rung of the ladder is unhealthy → degrade gracefully per surface and if nothing is viable, return a clean retryable error rather than silently worse output.
"Let me trace one Tab request end to end. It's admitted against a token bucket - over budget, I shed fast with a retryable error rather than queue it past its 100ms life. Routing picks a model and provider by config and live health, preferring an upstream with a warm prefix cache. The gateway serializes to that provider's dialect with a tight timeout and a hedge armed at the measured p95. Tokens stream straight back, deltas forwarded untouched for TTFT, tapped for metrics. If the provider 5xxs, it's retryable so I spend from the budget and cross-route to the next rung of the fallback ladder; if the circuit's open, I skip it entirely. An Agent request runs the same pipeline with the dials inverted: hedge rarely, fail over to an equivalent-tier model and propagate cancellation hard so an abandoned turn stops paying for tokens."
Drive the whiteboard with the request, not the boxes. Most candidates draw components and describe them statically. Animate a single request through the system and let the components reveal themselves as it hits each one - admission, routing, gateway, provider, stream - then run a failure through the same path. That narrative is the difference between describing a diagram and demonstrating you've operated one.