2 min lesson
What this means for provider-backed routing
Consider this situation: "You route to a third-party provider whose internal batching you can't control. Which set of levers is genuinely yours at the gateway?" Start with the decision, then the evidence.
Step 1 of 2
What this means for provider-backed routingyou don't own their batcher, but you own the dials in front of it
When you route to a third-party provider, their serving stack does the batching and you can't see inside it. What you control is upstream: concurrency limits, queueing and admission, hedging and which provider gets a given request. Those decisions interact with the provider's batch composition even though you never touch it.
- Concurrency limit
- How many in-flight requests you allow per provider/route
- Admission / queueing
- Whether a request enters now, waits or is shed under load
- Hedging
- Sending a duplicate to a second provider to cut tail latency, at extra cost
- Provider selection
- Which backend a request lands on, by health, latency, price, cache affinity
When you reach a batching decision in a design round, say the words "this is the throughput-versus-latency tradeoff" and then anchor it to the workload: "For Tab I'd cap batch size and concurrency to protect p99; for bulk Agent generations I'd allow larger batches because throughput per dollar matters more than a few ms of ITL." Tying the dial to the specific surface is the senior signal.
Don't claim continuous batching "removes" the latency cost of large batches. It removes the waiting and the head-of-line blocking of static batching; it does not repeal the fact that a bigger concurrent batch makes each decode step slower. Mixing those up is a common tell.