How to benchmark LLM inference honestly

A single number is not a result

Tokens per second, on its own, means almost nothing. The same model on the same GPU can differ ten times over depending on how many requests run at once, and by large amounts depending on how long the prompts and answers are.

A figure without its conditions cannot be repeated, compared or acted on. It is also usually flattering by accident. People test with short made-up prompts because they are convenient, and short prompts hide how much memory real work uses, so everything looks faster than it will be.

Measure a curve, not a point

Throughput depends on load, so the honest result is a sweep: run the same work with more and more requests at once (concurrency) and record what happens at each step. Three parts of that curve are worth reporting:

• The plateau: the point past which adding requests stops adding throughput. This is where a well-run system actually operates.
• The collapse: the point where requests start to fail rather than just slow down. It tells you how much room you have.
• The shape between them: whether the system slows gently or falls off an edge. For AI inference, which runs out of GPU memory, it is usually an edge.

Reporting only the peak hides two of the three things a reader needs. Reporting one arbitrary level of load hides all three.

The comparison trap

Benchmarks rarely mislead through made-up numbers. They mislead by comparing two things measured under different conditions. A made-up example shows how easy it is:

Read quickly, the cheaper GPU looks faster. That conclusion does not hold: the two runs had different numbers of requests running at once, the setting that most changes throughput. All you can honestly say is how each GPU behaved at its own load. To compare them, run the same sweep on both.

What a repeatable run records

A result should carry enough detail that a stranger could set up the same conditions. At least:

Model

The repository and exact version, not just the family name. "Latest" is not a version.

Precision

The format the model actually ran in, including any compression (quantization). It is often not the one in the model card’s headline.

Serving software

The server’s version and the settings it ran with. Settings change between releases and quietly change behaviour.

Hardware

The GPU model and count, the machine type, and whether it was spot capacity that could be taken back.

Workload

Prompt and answer lengths from your real data, as a spread, not the average of made-up prompts.

Load

How many requests ran at once, how many there were in total, and how long the measured stretch lasted.

Outcomes

Good answers, timeouts and errors, counted separately, not one success percentage.

Price basis

What the hardware cost per hour at the time, and whether startup time is inside or outside the figure.

Failed requests count against throughput

A run that shows a great tokens-per-second figure while dropping most of its requests did not achieve that throughput. It is a common and rarely deliberate mistake: the test tool reports on the requests that finished, and the ones that timed out quietly drop out of the count.

If most requests time out at a given load, the right summary is not a throughput number with a footnote. It is that the setup failed at that load. A test tool you can trust reports finished and failed requests side by side, and sorts the failures, because a timeout, a malformed answer and a server error mean three different things.

From throughput to cost per million tokens

The number that matters for a budget is cost per million tokens, and working it out is where optimism creeps in. The honest version is all the money spent divided by all the useful output:

cost per 1M tokens = (instance $/hr x billed hours) / (valid tokens produced / 1,000,000)

• Include startup time, or say plainly that you left it out. On short jobs it can be most of the bill.
• Count only tokens from good answers. Tokens from requests that later timed out were paid for and delivered nothing.
• Use the price you actually paid. Spot prices move; a figure worked out from a list price is a different claim.
• Say how many requests ran at once, because the same hardware at a different load gives a different cost.

Know your noise floor: temperature 0 is not repeatable

Before you compare two setups, find out how much the same setup disagrees with itself. Most people assume that at temperature 0 the answer is "not at all". It usually is not.

At temperature 0 the model always picks its top-scoring next token, so the choice is fixed once the scores are. But the scores themselves can move slightly. Computers add decimal numbers with tiny rounding errors, and the result depends on the order of the additions. A server running many requests together does the arithmetic in a different order depending on what else is running alongside yours. The difference is in the last few digits and normally invisible, until two candidate tokens score almost the same and the order flips. From that token on, the two answers can go completely different ways.

Hardware adds a second source, and it matters more on spot capacity. A different GPU model runs different low-level code for the same maths, so a job that is interrupted and re-run on another machine may not just be re-grouped with other requests; it may run different code. Any service that uses spot capacity has this property, whether or not it says so.

None of this makes benchmarking pointless. It makes one step essential:

1. Run your test set twice with identical settings and count how often the answers differ. That is your noise floor.
2. Run it again with a different number of requests at once. If the disagreement grows, the mix of requests running together affects your results, and you should record it.
3. Only then compare models or setups, and treat any difference smaller than the noise floor as no result at all.

This is the difference between results you can reproduce and results you can repeat byte for byte. You are not promising identical output on a rerun; you are promising that the same method on the same work lands in the same place, within a margin you have measured and stated. Claim byte-for-byte repeats and a reader with a terminal will disprove it in an afternoon.

Publish the method, not just the number

A benchmark exists so that someone else can decide whether to believe you. That only works if the method travels with the result: the workload, the setup, the caveats, and ideally the code that produced it.

The industry has learned this before. The authors of MLPerf, the standard industry benchmark, were blunt about why one was needed:

There is a clear need for industry-wide standard ML benchmarking and evaluation criteria.

Their answer is worth copying even at a much smaller scale: the benchmark "prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures". Not a number, but a set of rules under which numbers can be compared. The list of what to record above is that idea for a single team.

A modest number someone can reproduce is worth far more than an impressive one nobody can. That is also why the figures we quote about our own platform are conservative and come with their conditions.

Sources

Two efforts worth knowing about, for opposite reasons: one set a standard for measuring performance, the other showed how little of the ground evaluations covered once someone counted.