Why batching wins so much
A decode step spends most of its time moving data rather than doing arithmetic. To produce one token the accelerator must read the model's weights out of memory, and that read costs the same whether one sequence or thirty-two sequences are being advanced by it. Batching lets a single pass over the weights serve every sequence in the batch, so the marginal cost of an extra request in the batch is small: its share of the attention computation and its own key-value cache traffic. Throughput therefore climbs steeply as batch size grows, while the time for one step climbs gently, until you run out of memory or the arithmetic units saturate and the curve flattens.
That asymmetry is the entire economic case for a serving engine. Tokens per second per accelerator is what determines cost per million tokens, and it is a property of how well you keep the batch full, not of the model alone. An idle-ish server running one request at a time is paying full price for the weight reads and getting one token for them.
Static batching wastes the batch
The naive implementation waits for a fixed number of requests, runs them together, and returns when all are finished. This fails badly for generation because sequences have wildly different output lengths. A batch of eight where one request generates nine hundred tokens and seven generate forty means seven slots sit occupied and useless for the rest of the run, and none of the eight users get a response until the longest one finishes. You have converted a throughput mechanism into a latency amplifier.
Continuous batching, the scheduling model used by modern serving engines, works at the granularity of a single decode iteration instead. The scheduler holds a set of running sequences; after every step it retires any that have finished and admits waiting requests into the freed slots. Nobody waits for the slowest member of their cohort, and the batch stays close to full even under uneven load. This is why "we batch" is not a sufficient answer: static and continuous batching have opposite effects on user-perceived latency.
The KV cache is what limits batch size
Each sequence in the batch carries its own key-value cache, which grows with the number of tokens it has seen. Batch capacity is therefore bounded by memory rather than by a number you pick: long contexts mean fewer concurrent sequences, and a workload with hundred-thousand-token prompts may only sustain a handful. Paged allocation of that cache — the technique vLLM introduced — reduces the fragmentation and over-reservation that otherwise force you to size for the worst case, which is what lets the batch run larger in practice.
The consequence for capacity planning is that batch size is a dependent variable. If your prompts double in length, your achievable concurrency drops and your queue lengthens, without any change in request rate. Anyone treating maximum batch size as a tuning knob independent of context length will be surprised by a traffic mix shift.
What the throughput gain costs
The gains are real, and they are paid for out of tail latency. Three mechanisms do it, and naming them is what distinguishes a serving answer from a marketing one.
A fuller batch makes every decode step slightly slower, so inter-token latency for an individual user degrades as utilisation rises. That is the gentle part. Worse, prefill and decode compete: when a request with a very long prompt is admitted, the prefill pass monopolises the device, and every user already streaming sees a visible stall. Engines mitigate this by splitting prefill into chunks interleaved with decode steps, which smooths the stall without removing the underlying contention. And once demand exceeds what the running batch can hold, the excess waits in a queue, so time-to-first-token acquires a queueing component that is invisible at low load and dominant at high load.
The shape of this is what matters for an SLO. Throughput and tail latency trade against each other along a curve, and pushing utilisation towards the point of maximum throughput puts you on the steep part of that curve where a small traffic increase produces a large p99 increase. Choosing an operating point below maximum throughput is not waste, it is the purchase of predictability.
Why interactive and bulk traffic want separate pools
Put a nightly document-classification job and a live chat feature on the same endpoint and the batch fills with the job. The chat requests still get served, but they queue behind admission decisions made for work that nobody is watching, and their p99 becomes a function of how much backlog the batch job has. The two workloads want opposite configurations: the interactive pool wants headroom, modest batch limits and short queues; the bulk pool wants the batch saturated, large limits and a deep queue, because for it a minute of queueing is free.
Separating them can be physical, with distinct deployments, or logical, with priority classes and separate admission limits inside one engine, and either is defensible. What is not defensible is a single undifferentiated queue plus an SLO that only one of the two workloads cares about. If you are on a hosted API rather than your own serving stack, the same reasoning applies to your rate-limit allocation and your own client-side concurrency: bulk work should be the thing that gets throttled when you approach the limit, and that only happens if you built a way to tell the two apart. Providers' batch endpoints, which accept a job and return results asynchronously at a lower price, exist precisely to move that traffic off the path your users are waiting on.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).