1 min lesson
Open with budgets and volume
Put yourself in this case: "In the onsite design discussion, why would you route Tab completions and Agent steps differently through the same gateway?" Give the clearest next step.
Step 1 of 2
- 1Open with budgets and volume. State latency targets (sub-100ms for Tab), request volume, cost per request and a failure-rate budget before you draw boxes.
- 2Tie each surface to its profile. Tab is latency-critical and cheap; Agent is long and expensive; chat streams - they don't get the same routing.
- 3Make the tradeoff explicit. Hedging cuts tail latency but duplicates provider cost; say which way you'd lean and why.
- 4Close the failure loop. Circuit break an unhealthy provider, shed load before the fleet melts and degrade gracefully rather than failing hard.
“For Tab I'd budget sub-100ms and route to the fastest healthy provider, no hedging at first because the duplicate cost isn't worth it at that volume. For Agent, where a step already takes seconds, I'd hedge the slow tail and circuit-break a provider whose p99 spikes. Same gateway, different policy per surface.”
In “meet the team,” ask one question that only a real candidate asks: “When a provider browns out mid-stream, do you fail the request or fall back and how do you keep it idempotent under retry?” It shows you've thought about the actual hard part of their job.