1 min lesson
Scenario: cost is 30% over budget
Explain your answer to "Why is continuous (in-flight) batching a bigger throughput lever than simply increasing a fixed batch size?" Add one concrete detail from the lesson.
Step 1 of 2
Scenario: cost is 30% over budgetName the lever and the risk
- Lever
- Prefix / response caching
- How it cuts cost
- Repeat prefills and identical requests become hits
- UX risk to name
- Stale responses if invalidation is wrong; correctness over savings
- Lever
- Cheaper-model routing
- How it cuts cost
- Route price-sensitive surfaces to a smaller model
- UX risk to name
- Quality drop users can feel, especially in Agent
- Lever
- Better batching
- How it cuts cost
- Continuous batching refills freed slots; GPUs stay busy
- UX risk to name
- Fixed batches add head-of-line blocking; watch ITL/p99
- Lever
- Fallback ladder tuning
- How it cuts cost
- Prefer cheaper providers when quality headroom allows
- UX risk to name
- More failover hops add latency; cap the ladder depth
| Lever | How it cuts cost | UX risk to name |
|---|---|---|
| Prefix / response caching | Repeat prefills and identical requests become hits | Stale responses if invalidation is wrong; correctness over savings |
| Cheaper-model routing | Route price-sensitive surfaces to a smaller model | Quality drop users can feel, especially in Agent |
| Better batching | Continuous batching refills freed slots; GPUs stay busy | Fixed batches add head-of-line blocking; watch ITL/p99 |
| Fallback ladder tuning | Prefer cheaper providers when quality headroom allows | More failover hops add latency; cap the ladder depth |
Every cost lever has a latency or quality cost. Naming it is the senior move.
Catch your hand-waves
If you said "add caching" without saying what you cache, the hit-rate you'd expect or how you invalidate it, that's a hand-wave. If you said "route to a cheaper model" without naming which surface can absorb the quality hit, that's a hand-wave. Write each one down. They are your exact study gaps before the loop.