1 min lesson
The tradeoff you must name out loud
Use "The tradeoff you must name out loud" to tell the cases apart, then choose a response for each one.
Step 1 of 2
The tradeoff you must name out loudbigger batches are not free
Pushing batch size up raises throughput and GPU utilization - more tokens per dollar. It also raises per-request and tail latency, because each request now shares memory bandwidth and compute with more neighbors and the decode step for a larger batch takes longer. This is the throughput/latency tradeoff and serving research (vLLM, Sarathi-Serve) is largely about pushing what's new of it.
More throughput, higher GPU utilization
Lower cost per token
Worse p99 / ITL under load
Tighter, more predictable tail latency
Better for latency-critical Tab traffic
Lower utilization, higher cost per token
Sarathi-Serve's contribution is worth a sentence: it chunks long prefills and interleaves them with ongoing decodes so a big prefill doesn't stall everyone's token stream. The general principle is that prefill and decode compete for the same GPU and how you schedule them against each other shapes tail latency.