1 min lesson
Batching and the throughput/latency tradeoff
Respond to "What is the core difference between static and continuous (in-flight) batching and why does continuous batching reduce tail latency?" Give the answer first and the evidence second.
Step 1 of 2
Batching is how a GPU serves many users at once instead of one. How you batch is the throughput-versus-latency knob at the heart of serving - name it explicitly and the design round opens up.
GPUs are fastest when they process many sequences together, amortizing the cost of loading weights across the whole batch. The question is when you form the batch and what you do when sequences in it finish at different times.
- Approach
- Static batching
- How it works
- Wait until a full batch of requests arrives, run them together to completion
- Cost
- Latecomers wait for the batch to fill; the whole batch waits for the slowest sequence
- Approach
- Continuous / in-flight batching
- How it works
- Add and remove sequences every decode step; a finished sequence is swapped out and a queued one swapped in immediately
- Cost
- Far higher GPU utilization, much less head-of-line blocking
| Approach | How it works | Cost |
|---|---|---|
| Static batching | Wait until a full batch of requests arrives, run them together to completion | Latecomers wait for the batch to fill; the whole batch waits for the slowest sequence |
| Continuous / in-flight batching | Add and remove sequences every decode step; a finished sequence is swapped out and a queued one swapped in immediately | Far higher GPU utilization, much less head-of-line blocking |
Continuous batching (vLLM, TGI, TensorRT-LLM) is the modern default for exactly this reason.
The win from continuous batching is that the GPU never idles waiting for the slowest sequence and a short request stuck behind a long one doesn't wait for the long one to finish. Each decode step re-forms the working set, so capacity tracks demand step by step.