Fundamentals

Throughput vs. latency: why batch wins on cost

Latency and throughput pull in opposite directions. Pick the wrong one to optimize and you overpay by a multiple.

7 min read Updated August 19, 2026

Two different questions

Latency: How long a single request takes from submission to result. The right metric when a human is waiting.
Throughput: How many requests (or tokens) the system completes per unit of time per GPU. The right metric when you care about cost across a large job.

A system tuned for the lowest possible latency keeps spare capacity idle so any incoming request can be served immediately. A system tuned for the highest throughput keeps every GPU as busy as possible. You cannot fully maximize both at once.

Why they trade off

GPUs are massively parallel. Serving a single request leaves most of their compute idle, waiting on memory transfers. The fix is batching: running many requests through the GPU together so its arithmetic units stay saturated.

Batching raises throughput dramatically — but it adds a little latency to each individual request, because requests wait briefly to be grouped and share the GPU. For interactive use that added latency is unacceptable. For batch use it is free: nobody is waiting, so trading a few seconds of per-request latency for several times the throughput is pure savings.

This is not a soft tradeoff that better engineering eventually dissolves. It has a specific mechanical cause, stated plainly in the serving-systems literature:

Batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.
— Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024)(opens in a new tab)

The reason is that a request has two phases with opposite appetites. The same paper puts it as directly as anyone has: prefill iterations "have high latency but saturate GPU compute", while decode iterations "have low latency but also low compute utilization" — which is what makes "batching highly effective for decodes and consequently for overall throughput".

So a system serving both phases at once must choose whom to disappoint. Interactive serving protects the waiting user and leaves compute on the table; batch serving fills the machine and lets any individual request wait its turn. Research since has attacked the conflict from both ends — splitting prefills into chunks to stop them stalling decodes, or running the two phases on separate GPUs entirely — but the tension itself is structural, not a bug awaiting a fix.

Utilization is the hidden bill

An interactive endpoint provisioned for peak demand might average 20-40% GPU utilization — you pay for 100% of the hardware but use a fraction of it. The unused capacity is the cost of readiness.

A batch system aims for near-full utilization: queue depth is a feature, not a problem, because there is always more work to pack onto the GPU. Higher utilization spreads the fixed hourly cost of the hardware across more useful work, which is the same thing as a lower cost per token.

Choosing which to optimize

The decision is not technical, it is about the work: is a human blocked on each result? If yes, optimize latency and accept the cost of readiness. If no, optimize throughput and capture the savings. The mistake most teams make is running clearly-batch workloads on latency-optimized endpoints out of habit, paying interactive prices for work that never needed them.

Sources

If you want the tradeoff in its primary form rather than ours, these three are the ones to read. They agree on the diagnosis and differ on which side of it they are optimizing.

Sources and further reading

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve(opens in a new tab)
Agrawal et al., OSDI 2024. Names the prefill/decode interleaving as the cause and splits prefills into chunks to build stall-free schedules; reports 2.6x-5.6x higher serving capacity depending on model and hardware.
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving(opens in a new tab)
Zhong et al., OSDI 2024. The other answer to the same problem — put the two phases on different GPUs so they stop interfering at all.
Orca: A Distributed Serving System for Transformer-Based Generative Models(opens in a new tab)
Yu et al., OSDI 2022. The origin of continuous (iteration-level) batching, which is what makes high-throughput serving possible in the first place.
What a request really costs
Turning the throughput number into a cost per million tokens.

Frequently asked questions

Does optimizing for throughput make my results slower?

It can add a little latency to each individual request, but the whole job often finishes sooner and far cheaper because the hardware is used efficiently. For batch work, where no one is waiting on a single result, that trade is almost always worth it.

Related guides

What is batch AI inference?

Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.

What an LLM inference request really costs

Behind every per-token price is a GPU running for some number of seconds. The GPU-seconds model of inference cost, and the two levers that actually move it.

GPUs for AI inference: T4, L4, L40S, A100, H100

The GPU you serve a model on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.