Throughput vs. latency: why batch wins on cost
Should an AI system answer each request fast, or get the most work done per dollar? It cannot fully do both. Work that has nobody waiting should choose the second, and paying for the first instead can cost several times as much.
Two different questions
- Latency
- How long one request takes, from sending it to getting the answer. What matters when a person is waiting.
- Throughput
- How much work each GPU finishes per hour, in requests or tokens. What matters when you care about the cost of a large job.
A system built for the fastest answers keeps spare machines idle, so any new request can start at once. A system built for the most work per dollar keeps every GPU as busy as possible. You cannot have both fully at the same time.
Why they trade off
A GPU does thousands of calculations at once. Serving a single request leaves most of that capacity idle, waiting for data to arrive from memory. The fix is batching: running many requests through the GPU together, so its calculating units stay busy.
Batching multiplies throughput, but each request takes a little longer, because it waits to be grouped and then shares the GPU. For a chat reply that delay is unacceptable. For batch work it costs nothing: nobody is waiting, so giving up a few seconds per request for several times the throughput is pure saving.
Better engineering will not make this trade go away. It has a specific mechanical cause, stated plainly in the research on serving models:
Batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.
The cause is that each request has two phases with opposite needs. "Prefill" reads the whole prompt at once; "decode" writes the answer one token at a time. The same paper puts it directly: prefill iterations "have high latency but saturate GPU compute", while decode iterations "have low latency but also low compute utilization", which is what makes "batching highly effective for decodes and consequently for overall throughput".
So a system serving both phases must choose whom to disappoint. Interactive serving protects the waiting user and leaves capacity unused; batch serving fills the machine and lets each request wait its turn. Researchers have narrowed the gap from both sides, by splitting prompts into chunks so they stop holding up answers, or by running the two phases on separate GPUs. The tension itself remains: it is built into how these models run.
How batch size affects latency and throughput
Raising the batch size (how many requests a GPU runs at once) raises throughput steeply and latency only a little, up to a limit set by the GPU’s memory. Past that limit throughput stops rising and latency climbs fast, until requests start to fail.
| Batch size | Throughput | Latency per request |
|---|---|---|
| One request | Lowest: most of the GPU waits on memory | Lowest: nothing shares the GPU |
| Growing, within memory | Rises steeply: one read of the model serves every request in the batch | Rises slowly: each step does more work |
| Near the memory limit | Flattens: the cache is full, so new requests wait to start | Rises fast: time spent queued adds to time spent running |
| Past the limit | Falls: requests are evicted, rebuilt and retried | Requests time out and fail rather than finish slowly |
So the right batch size depends on who is waiting. A chat service keeps batches small enough that each answer stays fast. Batch work runs as large a batch as memory allows, just short of the limit, because nobody notices a request that takes a few seconds longer.
Utilization is the hidden bill
An interactive service sized for its busiest moments might keep its GPUs busy only 20–40% of the time on average. Someone pays for all of the hardware while a fraction of it does work. That idle share is the cost of being ready.
A batch system aims to keep GPUs almost always busy. A long queue is a good thing, because there is always more work ready to run. A GPU costs the same per hour however busy it is, so the more work it does in that hour, the less each token costs.
That is the case for sending a batch service all of your work at once. A whole backlog keeps every GPU full from the moment it starts; the same work sent a few requests at a time leaves machines half-empty or waiting for enough to be gathered. On Otium you do not need to pace it at all: pick a deadline and send everything, and running the queue is our job.
Choosing which to optimize
The choice depends on the work, not the technology. Is a person waiting on each result? If yes, pay for fast answers. If no, pay for throughput and keep the savings. The common mistake is running work that could wait on fast, interactive services out of habit, and paying interactive prices for it.
The same holds for every step of a pipeline. Each step of a multi-step job is batch work if nobody waits on it, and a deadline for the whole job, rather than one per step, gives the scheduler the most room to keep machines busy.
Sources
To read the trade-off at the source, start with these three papers. They agree on the cause and differ on which side of it they improve.
Frequently asked questions
What is the impact of batch size on latency vs throughput?
A larger batch raises throughput steeply and latency a little, until the GPU’s memory is full. Past that point throughput stops rising, latency climbs fast and requests start to fail. Interactive serving keeps batches small; batch work runs them as large as memory allows.
Does optimizing for throughput make my results slower?
Each request may take a little longer, but the whole job often finishes sooner and costs far less, because the hardware is kept busy. For batch work, where no one is waiting on a single result, that trade is almost always worth it.
Related guides
Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
The GPU a model runs on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.