Estimating what an AI job will cost before you run it

The basic sum

For one step, the cost is the number of items times what one item costs. What one item costs is its input tokens times the input price, plus its output tokens times the output price. Output tokens usually cost several times more than input tokens, so a step that writes a lot costs more than one that reads a lot.

If tokens are new to you, our guide on tokens and pricing covers them first. The rest of this guide is about getting the inputs to that sum right, and the costs it leaves out.

Measure token counts, do not guess them

Rules of thumb (about four characters of English per token) are fine for a first guess, but a real estimate should start from your data:

1. Take a sample of a few hundred real items, not hand-picked ones.
2. Build each item’s full prompt, including the instructions, examples and any reference data you send with it.
3. Count the tokens with the tokenizer of the model you plan to use. Different models split text differently.
4. Run the sample through the model and count the output tokens too. How much a model writes depends on the model and the prompt, and is hard to guess.

Write down the typical value and a high one, such as the 95th percentile: the length that 95 items in 100 stay under. You will need both.

What the basic sum leaves out

Retries

An answer that fails a check and is asked for again is paid for twice. Multiply the step’s cost by one plus its retry rate.

Escalation

Items moved to a bigger model after failing on a small one are paid for on both. Add the share that moves up, at the bigger model’s price.

Branches

A step that runs only for some items costs only for those. Multiply by the share of items that reach it, which is a saving the basic sum misses.

Per-call fees

Lookups in other services, search or data APIs, are often charged per call, not per token. At thousands of items, a small fee per call can rival the model cost.

Reruns

If you change a prompt and run the job again, you pay again. Budget for at least one more run than you plan.

A worked example

Take 10,000 support tickets through three steps. Every ticket is classified on a small model. Billing tickets, about 30%, get a refund check on the small model and one order lookup in a service that charges $0.002 per call. Every ticket gets a reply drafted on a medium model.

The prices are illustrative: the small model at $0.10 per million input tokens and $0.40 per million output tokens, the medium one at four times that.

Three things stand out. The order lookup, at a fifth of a cent, costs more than everything on the small model put together. The retries and escalation are small, because the step they apply to is cheap. And the reply, the one step on the bigger model that writes the most, is over half the bill. That is where to look first if the total is too high.

Plan for the expensive case

Averages hide the tail. If the replies run long, at the 95th-percentile length of 600 output tokens instead of 300, the reply line rises by 10,000 × 300 × $1.60 per million, or $4.80, and the job costs about $20.86. Budget for that number, not $16.06.

Set a hard limit on output length (max_tokens) for each step. It caps the worst case, and a step that hits the cap often has a prompt problem worth fixing anyway.

Check the estimate with a pilot run

Before the full job, run 1% or so of it and compare the bill with your estimate, line by line. If they differ by more than a little, find out why before you run the rest. The usual causes are output lengths, a retry rate that is higher than expected, or a prompt that is longer than it looked. A pilot that costs $0.16 is a cheap way to avoid a surprise at $16, or at $16,000.

Then put a guardrail in place: a spending limit on the account, or a check in your own code that stops submitting work once a budget is reached. An estimate tells you what should happen; a limit decides what can.

Cost estimates on Otium

In an Otium workflow, each step can declare the tokens it expects with estimate: { tokens_in, tokens_out }. The compiler adds up the steps and reports an estimated cost per item and per 10,000, and the typical and slow-case duration, before anything runs. It is a guide to compare versions of a workflow, not a quote.

When a batch is accepted, each run sets aside its slow-case cost from your balance, so a run never starts that your balance cannot finish, and a monthly spending limit holds new work once it is reached. Every attempt that returns an answer is charged at its step’s rate, including one a check rejects, and a call to your own endpoint from a tool step costs $0.002.

Repeated input can bring the real bill below the sum. When a prompt starts with text that has just been processed, such as the instructions every item shares, that part is served from cache, and on Otium cached input is free. Items that arrive together share the cache, so send the whole job at once rather than a few items at a time: more of its input is free, and a batch big enough to fill a machine starts sooner.

Further reading