The concurrency cliff: why inference throughput collapses instead of slowing down

Why you raise concurrency in the first place

Answering one request at a time wastes most of a GPU. To produce each word, the card reads the whole model out of its memory and then does relatively little calculation with it. With a single request, the calculating parts of the chip sit mostly idle while the model streams past.

Run many requests at once (concurrency) and that same read serves all of them. Each request runs at about the same speed, but the total output climbs steeply, and the total is what sets your cost per million tokens. This is why batch AI is cheaper than real-time AI at the hardware level, before any pricing or scheduling comes into it.

Where the ceiling comes from: KV cache, not FLOPs

It is tempting to assume the limit is computing power: that at some point the GPU simply runs out of calculations. That is almost never what happens first. The limit is memory, and in particular the KV cache (explained below). The paper that introduced vLLM, a widely used serving program, states the problem in its first sentences:

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size.

Note the last three words: limiting the batch size. Memory sets how many requests can run at once, which sets throughput, which sets your cost per million tokens. The paper’s technique, PagedAttention, wastes less memory and reports two to four times the throughput for the same response time. It raises the ceiling; it does not remove it.

Here is the KV cache. As a model writes, it keeps a working record of every token it has seen so far in that request. That record lives in the GPU’s memory next to the model, and it grows with both the number of requests running and the length of each one. The card’s memory splits roughly three ways:

Model weights

Fixed. Set by the model’s size and how it is compressed. Paid once, whatever the load.

KV cache

Variable. Grows with the number of requests running times the tokens in each. This is the part that runs out.

Working space

Room for the calculations themselves, plus the serving software.

So how many requests you can safely run depends as much on your prompts as on your hardware. Long prompts with short answers use the cache very differently from short prompts with long answers, and a card that comfortably runs thirty of one may fail at eight of the other.

What going over the edge looks like

Below the edge, everything works: every request finishes, and total throughput is many times what a single request gets. A few steps past it, the same card running the same work does not get slower. It stops finishing most requests at all, because they run out the clock waiting.

We have seen this in our own testing: one concurrency setting separated a run where every request finished from one where almost none did. The jump is that sharp.

Why it is a cliff and not a slope

Serving software only starts a request if it can reserve room in the cache for it. Past the memory limit, it can no longer keep every started request in memory, so it starts evicting some, rebuilding their cache when they resume, and sending them to the back of the queue.

That churn costs time on a system that is already full. The queue grows faster than it drains, requests hit their time limits, and the run collapses into mass failure. It does not degrade gracefully, because the thing it ran out of cannot be split: a request either has room for its cache or it does not.

This is also why there is no warning zone between healthy and failing. You cannot find the edge by watching a dashboard in production; you have to go looking for it on purpose.

Finding your own ceiling

The only reliable method is a sweep: run the same realistic workload at rising concurrency and record what happens at each step.

1. Use real prompts. Short made-up prompts will mislead you, because they barely use the resource that runs out.
2. Double the concurrency each step (1, 2, 4, 8, 16 …) rather than creeping up: the interesting range is wide, and the low steps are slow to measure.
3. Record finished requests and time-outs at every step, not just throughput. A throughput figure from a run that dropped most of its requests means nothing.
4. Watch for the plateau: where total throughput stops improving is your practical setting, and it comes before the failure point.
5. Then find the failure point anyway, so you know how much margin you have.

How the inference engine changes the ceiling

Every serving engine runs into the same memory limit, but each sets it with different settings and behaves differently when it is reached. Some admit as many requests as memory allows and push some back out when it runs short; others admit only what they can promise to finish. The first gets more out of the card and is the one that falls off the cliff; the second never collapses and leaves some throughput unused.

Two consequences follow. A setting tuned on one engine does not carry over to another, even for the same model on the same card, because they divide memory and count requests differently. And the engine’s policy decides what the cliff looks like: an engine that evicts shows the collapse described above, while one that refuses or holds back shows a lower plateau and rejected requests instead.

State-space and hybrid models

Some newer models replace part of their attention with layers that keep a fixed-size state per request (Mamba-style or linear attention). That state is neither weights nor KV cache, and an engine may reserve it for every request slot up front. The request cap then costs memory before a single request arrives. We met this ourselves: on a smaller card, vLLM refused to start one such model with “max_num_seqs (256) exceeds available Mamba cache blocks (113)”. The same model needs a lower request cap on a smaller card, so the right setting depends on both the model and the GPU.

What this means if you are buying batch inference

The ceiling moves with the model, its compression, the card and the shape of your prompts. Change any of them and the safe concurrency changes too, which is why running your own batch AI is less a one-off setup than continuous tuning.

It is also why cheap hardware and high throughput are linked. A cheaper card with less memory does not just run more slowly; it runs fewer requests at once before it stops working. Choosing the right setting for each model and workload is most of what separates a good cost per million tokens from a bad one, and it is work a managed batch service does for you.

That includes deciding how many of your requests run at once. You do not need to pace your work to protect the GPU, or keep a rate limiter of your own. On Otium, send the whole batch at once: the service sets how many requests each card runs, keeps the rest queued safely, and retries anything that fails.

Sources

The mechanism is well documented, and worth reading in the original if you are going to run near the edge.