Tokens, explained: how AI inference is priced

What a token is

A token is a chunk of text — roughly a word-piece. Models do not read characters or whole words; they read tokens produced by a tokenizer that splits text into common subword units. Frequent words are often a single token; rarer words and code split into several.

A useful rule of thumb for English prose is about 4 characters per token, or roughly 0.75 words per token — so ~750 words is about 1,000 tokens. Code, JSON, non-English text, and unusual formatting tokenize less efficiently (more tokens per character), so always measure rather than guess for production estimates.

Input vs. output tokens

Every request has two token counts, and they are priced differently:

Input (prompt) tokens

Everything you send in — system prompt, instructions, context, the item being processed. The model reads these in a single, highly parallel "prefill" pass, which is cheap per token.

Output (completion) tokens

Everything the model generates. These are produced one at a time ("decode"), which is slower and more expensive per token — often several times the input rate.

This is why output tokens usually cost more than input tokens on a published rate card: generation is the bottleneck. It also means the shape of your workload matters. A job with long prompts and short answers (classification, extraction) is cheap; a job with short prompts and long answers (open-ended generation) is dominated by output cost.

Estimating the cost of a job

To estimate a batch, multiply per-item token counts by item count, then by the per-token rates:

cost = items x (avg_input_tokens  x input_rate_per_token
             + avg_output_tokens x output_rate_per_token)

Worked example: enriching 100,000 product listings at roughly 800 input and 400 output tokens each is 80M input + 40M output tokens. At an illustrative $0.20 / 1M input and $0.80 / 1M output, that is $16 + $32 = $48 for the run. Change the model, the tier, or the prompt length and the number moves — but the method is always the same.

What token pricing hides

A per-token rate card is convenient, but it bundles together several different things: the raw compute, the model license, the provider margin, and the cost of always-on readiness. On an interactive proprietary API, the model-license and readiness components can dominate.

Batch inference on open models unbundles this. You pay for GPU-seconds of actual compute, not a per-token license on someone else proprietary weights, and not for idle readiness you are not using. The token is still a handy unit for quoting and comparison, but underneath, the real cost driver is how many GPU-seconds your job consumes — which is the subject of the next guide.

Where the token came from

It is worth knowing that the token is not a natural unit of language. It is an engineering compromise, and the specific compromise every modern model uses traces back to a 2015 machine-translation paper that borrowed a data-compression algorithm to solve a vocabulary problem.

encoding rare and unknown words as sequences of subword units

That is the whole mechanism, and it explains the billing quirks. A tokenizer is tuned on a corpus, so text that looks like its training data compresses well and text that does not — code, JSON, non-English scripts, unusual formatting — fragments into more tokens per character. You are not being charged more for JSON out of spite; the tokenizer genuinely sees more units. It also explains why the same paragraph costs different amounts on different model families: they were fitted to different corpora.