Tokens, explained: how AI inference is priced

What a token is

A token is a small chunk of text, often a word or part of one. Models do not read letters or whole words; a tokenizer first splits the text into common pieces. Frequent words are usually one token; rarer words and code split into several.

A rule of thumb for English prose is about 4 characters, or three quarters of a word, per token: 750 words is about 1,000 tokens. Code, JSON, other languages and unusual formatting take more tokens for the same length, so measure your real text before you rely on an estimate.

Input vs. output tokens

Every request has two token counts, and they are priced differently:

Input (prompt) tokens

Everything you send: the system prompt, instructions, context and the item itself. The model reads all of it at once in one pass (called "prefill"), which is cheap per token.

Output (completion) tokens

Everything the model writes back. It writes one token at a time (called "decode"), which is slower, so each output token costs more: often several times the input rate.

Writing is the slow part, which is why output costs more on almost every price list. It also means the shape of your work matters. Long prompts with short answers, like classifying or extracting, are cheap. Short prompts with long answers, like writing articles, cost mostly output.

Estimating the cost of a job

To estimate a batch, multiply each item’s token counts by the number of items, then by the price per token:

cost = items x (avg_input_tokens  x input_rate_per_token
             + avg_output_tokens x output_rate_per_token)

For example: enriching 100,000 product listings at about 800 input and 400 output tokens each is 80 million input and 40 million output tokens. At an illustrative $0.20 per million input and $0.80 per million output, that is $16 + $32 = $48 for the run. Change the model, the deadline or the prompt and the number moves, but the method stays the same.

On Otium, the shared start of a prompt is served from cache once the first item has run, and cached input is free: you pay for new input and for output. Items share that cache when they arrive together, so send the whole job at once, as one batch, rather than a few requests at a time. The more arrives together, the more of your input costs nothing.

A job with several steps is the same sum, done once per step: each step has its own prompt, its own answer length and its own model size. An Otium workflow does this arithmetic for you. Each step states the tokens it expects, and before anything runs you see the typical and worst-case cost of the whole pipeline.

What token pricing hides

A price per token is convenient, but it lumps several costs together: the computing itself, the model maker’s cut, the provider’s margin, and the cost of keeping machines ready for instant answers. On an interactive service selling its own model, the last two can be most of the price.

Batch work on open models (models anyone can download and run) pulls these apart. What you pay for is GPU time actually spent on your work, not a licence fee on someone else’s model and not idle machines kept waiting. The token stays a handy unit for quoting and comparing, but underneath, cost is driven by how many seconds of GPU time your job uses. That is the subject of the next guide.

Where the token came from

The token is not a natural unit of language. It is an engineering compromise, and the one every modern model uses goes back to a 2015 translation paper that borrowed a data-compression trick to handle rare words.

encoding rare and unknown words as sequences of subword units

That explains the billing quirks. A tokenizer is fitted to a body of text, so text like it splits into few tokens, and text unlike it (code, JSON, other scripts, odd formatting) splits into more. JSON is not charged more out of spite; the tokenizer really does see more pieces. It is also why the same paragraph costs different amounts on different model families: their tokenizers were fitted to different text.