2 min lesson
Scale, economics and what 'good' means here
Use the lesson to respond to this: "A provider starts returning 429s under your traffic. Is that a failover problem or a capacity problem and what does the distinction change?" Keep the answer plain.
Step 1 of 2
The role requires comfort with "cost/performance tradeoffs at scale: GPU utilization, provider economics, capacity planning." Routing decisions move a real bill, so you need to talk about money and capacity as fluently as you talk about latency.
- Per-token pricing
- Input and output tokens priced separately; output usually costs more. Routing to a cheaper model directly lowers cost per request.
- Rate limits
- Providers cap you on TPM (tokens/min) and RPM (requests/min). Hit the cap and you're throttled - a capacity problem, not an outage.
- Committed capacity
- Reserved/provisioned throughput trades a fixed spend for guaranteed headroom; on-demand is flexible but rate-limited and pricier at peak.
- GPU utilization
- For self-hosted inference, idle GPUs are pure waste; batching raises throughput per GPU but adds queueing latency.
The context makes efficiency existential, not nice-to-have. Anysphere runs a very small team against very high ARR, so output per engineer and per GPU dollar is the whole game. A routing change that shaves cost per request at millions of requests is real money and "handle millions of requests" is the baseline expectation rather than a stretch goal.
- Lever
- Prompt / prefix caching
- Effect on cost
- Skips recompute on shared prefixes
- Effect on latency / quality
- Big latency win on cache hit; correctness risk if stale
- Lever
- Aggressive hedging
- Effect on cost
- Higher cost - you pay for the loser
- Effect on latency / quality
- Buys tail latency and reliability; per-surface call covered in the four-way tradeoff
| Lever | Effect on cost | Effect on latency / quality |
|---|---|---|
| Prompt / prefix caching | Skips recompute on shared prefixes | Big latency win on cache hit; correctness risk if stale |
| Aggressive hedging | Higher cost - you pay for the loser | Buys tail latency and reliability; per-surface call covered in the four-way tradeoff |
Levers beyond the pricing basics above - each still has a latency or quality side effect.
Define "good" in SLO terms and be ready to put numbers on it. Vague answers about "making it fast" lose to a candidate who says "p99 under X for Tab, four-nines availability on the gateway and a cost-per-request budget I'd track per surface."