What an LLM inference request really costs

The short answer: list prices per million tokens

Model APIs charge per million tokens, with separate prices for input (what you send) and output (what the model writes). At list prices that runs from $0.10 input and $0.50 output for the smallest models to $10.00 and $50.00 for the largest. Input the provider has just seen, such as the same instructions at the start of every request, is served from a cache at about 10% of the input price, and some vendors charge extra to write the cache. Work that can wait up to a day costs half, as batch.

Those are prices, with each vendor's costs and margin inside them. The compute underneath is a different number: in the worked example below, with illustrative figures, serving a million tokens costs about seven cents. What the GPU costs per hour and how busy it stays move that figure several times over. The rest of this guide shows how.

The GPU-seconds model

Set the price list aside and the compute cost of inference comes down to one idea: a GPU runs for some time at an hourly price, and in that time it finishes some number of requests.

The number that ties it together is throughput: how much work a GPU finishes per second. It needs a precise definition, because "tokens per second" is quoted in at least three ways that do not compare.

In this guide, throughput is per GPU: requests finished per second across the whole setup, divided by the number of GPUs. A model spread over two GPUs that finishes 6 requests a second has a per-GPU throughput of 3. And it is measured on your own work: your prompt and answer lengths, with as many requests running at once (concurrency) and as fast a response as you will really need.

R = completed requests per second, per GPU (deployment rate / GPU count)
L = tokens per request (input + output, at your workload's lengths)

compute_cost_per_request = gpu_hourly_price / (R x 3600)
compute_cost_per_1M_tokens = gpu_hourly_price / (R x L x 3600 / 1e6)

The two lines are the same equation; the second divides by tokens. Two things move it: what you pay per GPU-hour, and how much useful work a GPU finishes in that hour. A per-token figure only compares with another measured at the same mix of input and output, which is why the per-request form is usually the safer one to budget with.

Throughput is the most sensitive variable

Compute cost per request moves in the opposite direction to throughput. Double R on the same hardware at the same price and the cost halves. That makes throughput the most valuable number to measure and to improve.

Throughput comes mostly from batching: running many requests through the GPU together so it stays busy. Writing each new token is slowed less by arithmetic than by moving the model and its working memory around the chip, so a GPU serving one request at a time spends most of its time waiting on memory. Serving dozens at once shares that memory traffic across all of them. No single request gets faster; the GPU just gets more of them done for the same work.

incremental inference ... is often slow, due to the memory-bandwidth cost of repeatedly loading the large "keys" and "values" tensors.

That is why the same GPU can report very different throughput, and why a number measured on someone else's workload rarely transfers to yours:

• Prompt and answer lengths. Reading the prompt (engineers call it prefill) handles many tokens at once and is cheap per token. Writing the answer (decode) produces one token per request per step, so long answers take most of the time. Long prompts and answers also use more GPU memory per request, which limits how many fit at once.
• How full the GPU stays. A measured peak only counts while the GPU is full. Gaps in the queue, the last few requests of a job finishing alone, or too little work to fill the GPU all lower the throughput you get per hour you pay for.
• How fast each answer must be. A target for how quickly the first token, or each later token, arrives limits how many requests you can batch, because a bigger batch makes each request in it wait longer. Research on serving calls the useful number goodput: the request rate a GPU can keep up while still meeting its speed targets. Interactive work is held to that limit; work with a deadline hours away mostly is not.

The other lever: what you pay for the GPU

The same physical GPU has very different hourly prices depending on how you buy it:

Interactive services can use spot capacity too. Many run partly on it, spreading requests across many machines of several types and regions, with spare machines and standard capacity to fall back on. When the cloud warns that it is taking a machine back, that machine stops taking new requests and finishes or hands off what it has.

The real difference is what an interruption costs. For an interactive request it is an error the person sees, or a slow retry, so interactive services need enough spare machines that it never shows. For batch work with a deadline, the request goes back in the queue and runs a few minutes later, and the deadline absorbs the delay. That is why batch work can lean much harder on the cheapest row of the table, and why spot prices combined with full, busy GPUs bring the cost per request down.

For more on how spot works, see the guide to spot compute. For what GPUs actually cost by type and region, see the monthly GPU price report.

From compute cost to total serving cost

The equation above prices the GPU time that produced results. You pay for more GPU time than that, and for some things that are not GPUs at all:

• Startup. Starting a machine, downloading the model and loading it into GPU memory is paid time that produces nothing. On a small job it can be most of the bill; on a large one it is spread across many requests.
• Idle time. A GPU waiting for work, or half-empty while a job's last requests finish, costs the full hourly price.
• Retries. A request that fails, times out, is interrupted partway or gives an answer that fails a check is paid for twice.
• Storage and networking. Request and result files, stored copies of the model, and data moved between regions or out to the internet.
• People. Monitoring, being on call, checking quality and upgrading models. These do not grow with each request, but they are real, and they are most of what a managed service charges to take off your hands.

Two longer-term choices lower the compute side: a smaller or compressed (quantized) model that still meets your quality bar, and a model trained further on your task so that a cheaper one can do the job. Both raise R for the same work.

For multi-step jobs, how the steps are run matters too. In an Otium workflow, a model stays loaded across the steps that use it, so a step does not pay to start a machine again; a bad answer is retried in place, seconds later, rather than in a new batch; and steps that do not depend on each other run side by side.

The startup and idle lines are also why Otium asks for all of your work at once. Starting a machine costs the same for ten requests or ten thousand, and a deep queue keeps the GPU full until the job is done. Otium waits until there is enough work to fill a machine before starting one, so a whole backlog starts sooner and runs cheaper than the same work trickled in a few requests at a time.

A worked example

Job

1,000,000 requests, each 1,000 input tokens and 200 output tokens

GPU price

$1.00 per GPU-hour (assumed, spot-like)

Measured R

4.0 completed requests per second per GPU, at this input/output mix and a batch-friendly concurrency

Fleet

8 GPUs, each spending 15 minutes starting up

Overheads

idle and drain time 5% of compute; 3% of requests re-run after failures or interruptions

compute       1,000,000 / 4.0 = 250,000 GPU-s = 69.4 GPU-h   $69.40
startup       8 GPUs x 0.25 h                     =  2.0 GPU-h    $2.00
idle / drain  5% of 69.4                          =  3.5 GPU-h    $3.50
retries       3% of 69.4                          =  2.1 GPU-h    $2.10
                                                    -----------   ------
GPU total                                           77.0 GPU-h   $77.00
storage + network (assumed)                                       $2.00
                                                                  ------
total serving cost (excluding operations)                        $79.00

Read the result three ways. Compute cost is $69.40, or $0.069 per 1,000 requests. The full serving cost is about $79, or $0.079 per 1,000 requests: overheads added 14%, before counting anyone's time. Per token, the job handled 1.2 billion tokens, so it cost about $0.066 per million tokens overall, or $0.40 per million output tokens if you divide by output alone. That last figure is six times as high only because it carries the input cost too, which is why per-token prices should be compared at the same mix of input and output.

Now change one assumption. Suppose a speed target kept so few requests running at once that R fell from 4.0 to 1.2 a second. Compute alone becomes 231 GPU-hours, $231 at the same price: more than three times the cost on identical hardware. That is the case for batch in one change: when nobody is waiting on each answer, you can run as fast as the hardware allows, not as fast as a speed target allows. The 8 GPUs in this example finish in under 10 hours, well inside a 24-hour deadline.

What a million tokens costs on OpenAI and Otium

A price per token is the sum above with a margin on top, quoted per million tokens and split into input (what you send) and output (what the model writes). Output costs several times as much as input because it is written one token at a time, which is the slow, GPU-heavy half of the work.

Compare at your own mix of input and output, as the worked example shows: a job that sends long prompts and gets short answers pays mostly the input price, and one that writes long answers pays mostly the output price.

More sources