Cost & economics

What an LLM inference request really costs

Token prices are the retail view. Underneath, the cost of inference is a GPU running for a number of seconds. Once you see it that way, every optimization makes sense.

9 min read Updated August 19, 2026

The GPU-seconds model

Strip away the rate card and inference cost reduces to one equation: a GPU (or several) runs for some duration at some hourly price.

job_cost   = gpu_count x gpu_hourly_price x hours_run
cost_per_1M = gpu_hourly_price / (throughput_tokens_per_sec x 3600 / 1e6)

The fundamental cost identity

The second line is the one to internalize. The cost to produce a million tokens is the GPU hourly price divided by how many millions of tokens that GPU produces per hour. Two levers, and only two: what you pay for the GPU, and how fast it produces tokens.

Throughput is the most sensitive variable

Cost per million tokens is inversely proportional to throughput (tokens per second). Double the throughput and you halve the cost — from the same hardware at the same price. That makes throughput the highest-leverage thing to optimize.

Throughput on a modern serving stack is driven mostly by batching: processing many requests through the GPU at once so its compute units stay busy. A GPU serving one request at a time is mostly idle, waiting on memory; a GPU serving dozens of concurrent requests can be near fully utilized. This is the technical reason batch workloads are cheaper — they make dense batching trivial.

The underlying asymmetry was named early, in the paper that introduced multi-query attention:

incremental inference ... is often slow, due to the memory-bandwidth cost of repeatedly loading the large "keys" and "values" tensors.
— Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019)(opens in a new tab)

Read that as a cost statement. Generating a token is dominated by moving data, not by arithmetic — so the arithmetic units are mostly idle while the weights stream past. Batching does not make any single request faster; it gets more requests out of the same unavoidable memory traffic. That is why throughput is the lever and why it is nearly free for work nobody is waiting on.

The other lever: what you pay for the GPU

The same physical GPU has wildly different hourly prices depending on how you buy it:

Purchase mode	Relative price	Trade-off
On-demand	Highest	Guaranteed, uninterruptible, available now.
Reserved / committed	Lower	Cheaper per hour in exchange for a long commitment.
Spot / preemptible	Lowest	Can be reclaimed with little notice; price varies by region and time.

The same hardware, priced by the guarantees attached to it.

Interactive workloads are stuck near the top of this table — they cannot tolerate a reclaim mid-request, so they pay for guarantees. Batch workloads can live at the bottom, riding spot capacity and simply re-queuing anything that gets interrupted. Combining low spot prices with high batched throughput is what drives the cost per million tokens down.

Do not forget model load and overhead

Real jobs carry fixed costs that the clean equation ignores: time to launch an instance, pull and load the model weights into GPU memory, and warm the server. For a tiny job these can dominate; for a large batch they amortize to nearly nothing. This is why batch economics favor larger jobs and warm, reused capacity, and why model-load time is a number worth measuring, not just throughput.

Two more structural levers lower the cost basis over time: quantization and right-sizing (serving a smaller or compressed model that still clears your quality bar), and task-specific fine-tunes that let a cheaper model do the job. Both reduce GPU-seconds per item, compounding with the spot-and-batching savings.

Sources

The GPU-seconds identity is arithmetic, but the claim that throughput is the sensitive term rests on measurable properties of how transformers run. These are the primary references.

Sources and further reading

Efficiently Scaling Transformer Inference(opens in a new tab)
Pope et al., 2022. Builds an analytical model of inference efficiency and reports a new Pareto frontier on the latency / model-FLOPS-utilization tradeoff — the formal version of "you cannot maximize both".
Fast Transformer Decoding: One Write-Head is All You Need(opens in a new tab)
Shazeer, 2019. The clearest early statement that decoding is bound by memory bandwidth rather than compute.
Efficient Memory Management for LLM Serving with PagedAttention(opens in a new tab)
Kwon et al., SOSP 2023. Why batch size — and therefore throughput, and therefore your cost per million tokens — is set by KV-cache memory management. Reports 2-4x throughput at the same latency.
The concurrency cliff
What happens when you push the throughput lever too far: not slower, but gone.
Tokens, explained
The retail view this guide sits underneath — what you are actually billed for, and why output costs more than input.

Frequently asked questions

What is the single biggest driver of inference cost?

Throughput — tokens produced per second per GPU. Because cost per million tokens is inversely proportional to throughput, doubling it halves your cost from the same hardware. Dense batching is the main way to raise it.

How do I lower my cost per million tokens?

Raise throughput (batch densely), lower the GPU price (use spot/remnant capacity within a flexible deadline), and reduce GPU-seconds per item (right-size or quantize the model, trim prompts, and reuse warm capacity to amortize model-load time).

Latency is how fast one request finishes; throughput is how much work you get per dollar. Why they pull against each other, and which one batch work should buy.

Cold starts: the hidden cost of short inference jobs

Before a GPU produces a token it must boot, install a serving stack and load weights. On short jobs that setup can outcost the inference, and most is removable.

How to benchmark LLM inference honestly

Most published inference numbers are not comparable to anything. What a defensible benchmark records, why you measure a curve, and the mistakes that void one.

The concurrency cliff: why inference throughput collapses instead of slowing down

Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.