The concurrency cliff: why inference throughput collapses instead of slowing down

Why you raise concurrency in the first place

Generating tokens one request at a time wastes most of a GPU. Decoding is limited by memory bandwidth rather than raw arithmetic: for every token, the card reads the model weights out of memory and does comparatively little maths with them. Run a single request and the arithmetic units idle while the weights stream past.

Run many requests concurrently and that same weight read serves all of them at once. The per-request speed barely changes, but the aggregate — the number that determines your cost per million tokens — climbs steeply. This is why batch inference is cheaper than interactive inference at the hardware level, before any market or scheduling cleverness is applied.

Where the ceiling comes from: KV cache, not FLOPs

It is tempting to assume the limit is compute — that at some concurrency the GPU simply runs out of maths. That is almost never what happens first. The binding constraint is memory, and specifically the KV cache. The paper that introduced vLLM opens by stating the problem in one sentence:

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size.

Note the last three words: limiting the batch size. Memory management is not a tidiness concern here — it sets the ceiling on how many sequences can run at once, which sets throughput, which sets your cost per million tokens. PagedAttention borrows virtual-memory paging to reduce the waste, and reports 2-4x throughput at the same latency for doing so. It raises the ceiling; it does not remove it.

As a model generates, it keeps the attention keys and values for every token it has seen so far. That cache lives in VRAM alongside the weights, and it grows with both the number of concurrent sequences and the length of each one. Card memory therefore splits roughly three ways:

Model weights

Fixed. Set by parameter count and quantization format. Paid once, regardless of load.

KV cache

Variable. Grows with concurrent sequences × tokens per sequence. This is the part that runs out.

Activations and overhead

Working space for the forward pass, plus the serving stack itself.

The practical consequence is that your safe concurrency depends as much on your prompts as on your hardware. A workload of long prompts with short answers consumes cache very differently from short prompts with long generations, and a card that comfortably serves thirty of one may fail at eight of the other.

What going over the edge looks like

Here is the failure from our own bring-up runs, processing 738 production-shaped requests (large prompts, small structured JSON outputs) against an 8B open model on real spot instances. The interesting pair is the last two rows: the same card, the same workload, one setting apart.

At concurrency 8 the 16 GB card was healthy: every single request completed, nothing timed out, and aggregate throughput was more than twenty times the single-stream decode rate. At concurrency 32, the same card on the same job lost 700 of 738 requests. Not slower. Gone.

Why it is a cliff and not a slope

Serving stacks admit a sequence only if they can allocate its cache blocks. Past the memory ceiling the scheduler can no longer keep every admitted sequence resident, so it starts preempting: evicting sequences, recomputing their cache when they resume, and re-queueing work behind the requests already in flight.

That thrash is not free, and it lands on a system that is already saturated. Queueing delay grows faster than the work drains, individual requests blow through their client timeouts, and the run collapses into mass failure. The system does not degrade gracefully because the resource it ran out of is not divisible — a sequence either has room for its cache or it does not.

This is also why the collapse is abrupt in configuration space. There is no gentle warning band between "healthy at 8" and "catastrophic at 32", which means you cannot find the edge by watching a production dashboard. You have to go looking for it deliberately.

Finding your own ceiling

The only reliable method is a sweep: run the same representative workload at increasing concurrency and record what happens at each step.

1. Use real prompts. Synthetic short prompts will tell you a comfortable lie, because they under-consume exactly the resource that runs out.
2. Step concurrency up geometrically (1, 2, 4, 8, 16 ...) rather than in small increments — the interesting region is wide, and low-concurrency points are slow to collect.
3. Record valid completions and timeouts at every step, not just throughput. Throughput on a run that dropped most of its requests is a meaningless number.
4. Watch for the plateau: the point where aggregate throughput stops improving is your practical operating point, and it arrives before the failure point.
5. Then find the failure point anyway, so you know how much margin you actually have.

What this means if you are buying batch inference

The ceiling moves with the model, the quantization, the card, and the shape of your prompts. Every time one of those changes, the safe concurrency changes with it — which is why self-hosting batch inference is less a one-time setup than an ongoing tuning job.

It is also why cheap capacity and high throughput are not independent problems. A cheaper card with less memory does not simply run more slowly; it runs a smaller number of sequences before it stops running at all. Picking the right operating point on that curve, for each model and workload, is most of what separates a good cost per million tokens from a bad one.

Sources

The numbers in this guide are our own bring-up runs, with their caveats attached. The mechanism they illustrate is well documented elsewhere, and worth reading in the original if you are going to operate near the edge.