2 min lesson
Inference serving & latency
Work through this case: "An infra lead proposes doubling batch sizes to cut cost per request. What is your immediate question and which metric would you watch before approving?" Say what you would do and why.
Step 1 of 3
Cost per inference is not a property of the model. It falls out of four serving decisions made together: model size, utilization, batching and routing.
On the systems round you will be expected to reason fluently about serving without writing kernels. The mechanism that ties throughput to cost is batching: the GPU runs many requests through one forward pass, so the fixed cost of that pass is shared. Bigger batches mean lower cost per request. They also mean a request may wait for the batch to fill, which adds latency. That tension is the heart of inference serving.
- Lever
- Larger batches
- Effect on cost/request
- Lower (shared forward pass)
- Effect on latency
- Higher (queue + bigger pass)
- Tension
- Throughput vs. responsiveness
- Lever
- Smaller / routed model
- Effect on cost/request
- Lower for simple queries
- Effect on latency
- Lower (less compute)
- Tension
- Quality vs. cost on hard queries
- Lever
- Higher utilization
- Effect on cost/request
- Lower (fixed cost amortized)
- Effect on latency
- Risk of contention at the edge
- Tension
- Efficiency vs. headroom
| Lever | Effect on cost/request | Effect on latency | Tension |
|---|---|---|---|
| Larger batches | Lower (shared forward pass) | Higher (queue + bigger pass) | Throughput vs. responsiveness |
| Smaller / routed model | Lower for simple queries | Lower (less compute) | Quality vs. cost on hard queries |
| Higher utilization | Lower (fixed cost amortized) | Risk of contention at the edge | Efficiency vs. headroom |
Every cost win has a latency or quality cost attached - name the tradeoff, do not pretend it is free.