How to benchmark LLM inference honestly

A single number is not a result

Tokens per second, on its own, is close to meaningless. The same model on the same card can differ by an order of magnitude depending on how many requests run at once, and by large factors depending on how long the prompts are and how long the outputs run.

A figure without its conditions cannot be reproduced, compared, or acted upon. Worse, it is usually flattering by accident: people benchmark with short synthetic prompts because they are convenient, and short prompts understate memory pressure enough to make everything look faster than it will be in production.

Measure a curve, not a point

Throughput is a function of load, so the honest artefact is a sweep. Run the same workload at increasing concurrency and record what happens at each step. Three features of that curve are worth reporting:

• The plateau — the concurrency beyond which aggregate throughput stops improving. This is your realistic operating point.
• The collapse — the concurrency at which requests begin to fail rather than merely slow. It tells you how much margin you have.
• The shape between them, which tells you whether the system degrades gently or falls off an edge. For memory-bound inference it is usually an edge.

Reporting the peak alone hides the two things a reader most needs. Reporting a single arbitrary concurrency hides all three.

The comparison trap, with a worked example

The most common way benchmarks mislead is not fabrication, it is comparing two things measured under different conditions. Our own first runs are a clean illustration, and we publish them with the caveat attached.

Read casually, the cheaper card looks faster. It is not a valid conclusion: the two runs were taken at different concurrencies, which is the single variable that most strongly determines aggregate throughput. The only defensible statement is that each card behaved a particular way at a particular load, and that a same-concurrency sweep across both is a run we still owe.

What a reproducible run records

A benchmark result should carry enough configuration that a stranger can rebuild the conditions. At minimum:

Model identity

Repository and exact revision, not just a family name. "Latest" is not a version.

Precision and quantization

The serving format actually loaded, which frequently is not the one in the model card headline.

Serving stack

Server version and the flags it ran with. Flags change between releases and silently change behaviour.

Hardware

Card model, count, and the instance type — including whether it was interruptible capacity.

Workload shape

Input and output token distributions from the real dataset, not averages of a synthetic one.

Load

Concurrency, total request count, and wall-clock duration of the measured window.

Outcomes

Valid completions, timeouts, and errors, classified — not a single success percentage.

Price basis

What the hardware cost per hour at the time, and whether cold start is inside or outside the figure.

Success rate is part of throughput

A run that produced a great tokens-per-second figure while dropping most of its requests did not achieve that throughput. It is a common and rarely deliberate error: the measurement harness reports on the requests that finished, and the ones that timed out quietly leave the denominator.

In one of our own runs, 700 of 738 requests timed out at high concurrency. The correct summary of that run is not a throughput number with a footnote — it is that the configuration failed. Any harness worth trusting reports completions and failures with equal prominence, and classifies the failures, because a timeout, a malformed output, and a server error mean three different things.

Turning throughput into cost per million tokens

The number that actually matters commercially is cost per million tokens, and deriving it is where optimism creeps in. The honest form is total money divided by total useful output:

cost per 1M tokens = (instance $/hr x billed hours) / (valid tokens produced / 1,000,000)

• Include cold start, or state clearly that you excluded it. On short jobs it can be most of the bill.
• Count only tokens from valid completions. Tokens generated by requests that later timed out were paid for and delivered nothing.
• Use the price you actually paid. Interruptible capacity prices move; a figure computed against a list price is a different claim.
• State the concurrency the run used, because the same hardware at a different load produces a different cost.

Know your noise floor: temperature 0 is not determinism

Before you compare two configurations, you need to know how much the same configuration disagrees with itself. Most people assume the answer is zero at temperature 0, and it usually is not.

Greedy decoding is deterministic given the logits: it takes the highest one. But the logits are not fixed. Floating-point addition is not associative — (a + b) + c does not always equal a + (b + c) — and a server using continuous batching processes your request alongside whatever else is in flight, changing the shape of the matrix operations and therefore the order the additions happen in. The result differs in the last bits. That is normally invisible, until two candidate tokens sit close together in probability and the ordering flips. From that token on, the two sequences can diverge completely.

Hardware is the other half of it, and it bites harder on interruptible capacity. A different GPU model selects different kernels, so a job that gets preempted and re-runs elsewhere is not merely re-batched — it is running different code. Any provider using spot capacity has this property, whether or not they mention it.

None of this makes benchmarking futile. It makes one step mandatory:

1. Run your evaluation set twice under identical settings and measure the disagreement rate. That is your noise floor.
2. Run it again at a different concurrency. If the disagreement grows, batch composition is a live variable for your workload and belongs in what you record.
3. Only then compare models or configurations — and treat any gap smaller than the noise floor as no result at all.

This is what separates a reproducible benchmark from a repeatable one. You are not promising that a rerun produces identical bytes; you are promising that the same method on the same workload lands in the same place, within a margin you have measured and published. Claim the first and a reader with a terminal will disprove it in an afternoon.

Publish the method, not just the number

The purpose of a benchmark is to let somebody else decide whether to believe you. That only works if the method travels with the result: the workload, the configuration, the caveats, and ideally the code that produced it.

This is an old lesson in systems measurement, and the industry has already institutionalised it once. The MLPerf authors were blunt about why a standard was needed at all:

There is a clear need for industry-wide standard ML benchmarking and evaluation criteria.

Their solution is worth stealing even at a much smaller scale: the benchmark "prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures". Not a number — a set of rules under which numbers become comparable. That is what the configuration list above is trying to be for a single team.

This is why we publish our own runs with their flaws attached — bring-up runs described as bring-up runs, an inflated cold-start figure labelled as inflated, non-comparable points labelled as non-comparable. A modest number somebody can reproduce is worth considerably more than an impressive one nobody can, and the difference in credibility compounds every time you publish again.

Sources

Two efforts worth knowing about, for opposite reasons: one standardised how performance is measured, the other showed how thin evaluation coverage was once somebody counted.