Prompt caching and batch economics
If every one of your 100,000 requests starts with the same long set of instructions, the model reads those instructions 100,000 times. Prompt caching lets it read them once and reuse the work, and batch jobs, which repeat the same instructions by design, gain the most from it.
What prompt caching is
Before a model can answer, it works through every token of the prompt and builds an internal summary of it (engineers call this the key/value, or KV, cache). Prompt caching keeps that work for the start of a prompt, its prefix, so the next request that starts the same way can reuse it instead of doing it again.
Only the shared start is saved: usually the system prompt, the instructions, any examples and any fixed reference text. The part that changes with each request, the item being processed, is still worked through from scratch.
Why batch work benefits most
Batch jobs repeat themselves by design. Enriching a product catalog means sending the same detailed instructions with each of thousands of products. The instructions might run to thousands of tokens and each product to a few hundred. Cache the instructions once, and every later product pays only for its own part.
How caching shows up in a bill
Providers that offer caching usually split the input price in two, so a request is billed in three parts:
- Uncached input
- Input tokens the model works through fresh: the first time, or after the cache expired. Full input price.
- Cached input
- Input tokens reused from the cache. Much cheaper, often a small fraction of the full price.
- Output
- The tokens the model writes. Caching does not change their price.
bill = uncached_input x r_uncached
+ cached_input x r_cached (r_cached << r_uncached)
+ output x r_outputA job where most input comes from the cache can see its input cost fall sharply. The catch: the provider decides what counted as cached, based on what happened to be in the cache when each request arrived. So the saving depends on something you do not fully control.
Otium does not split the price this way, because cached input is free. When a prompt starts with text that has just been processed, that part is served from cache and costs nothing; you pay for new input and for output. Pricing compares the two approaches.
Best-effort caching, and how batch does better
On interactive APIs, caching is best-effort. A cached prefix lives for a few minutes and can be dropped when the service is busy, so two identical requests a few minutes apart can be billed very differently.
Batch is friendlier. A batch service decides when each item runs, so it can run the items that share a prefix together, while that prefix is still in memory. That turns a lucky cache hit into something close to a sure one.
The technique has a name and a paper. SGLang introduced RadixAttention, which keeps finished requests' cached state in a tree so that a later request with the same prefix finds it on its own, without being told:
The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding.
They report up to 6.4 times the throughput of the systems they compared against. The key point: reuse only happens while the shared prefix is still in memory, which depends on when requests run. Two requests that share a 4,000-token start but run an hour apart share nothing. Running them together is worth more than any prompt trick, and only the service running the work can do it.
But a service can only run together the requests it has. That is your half of the bargain: send the whole job at once, as one batch file or one workflow run, and every item that shares a start arrives while that start can still be reused. Send it a few requests at a time and much of it arrives after the shared start has gone, so the same instructions are paid for again and again.
Shared reference data in a workflow
Much of the repeated text in a batch prompt is reference data: a style guide, a list of categories, a field list for each product type. In an Otium workflow, that data is published once as an asset instead of being copied into every item’s input, and it is placed into each prompt that uses it, the same for every item in the batch.
Sources
Caching is the clearest case of knowing what is coming next being worth money. That is why it also appears in the research on optimizing a whole workload rather than one request at a time.
Frequently asked questions
What gets cached in prompt caching?
The shared start of your requests: usually the system prompt, instructions and fixed examples. The model reuses the work it did on those tokens instead of doing it again. The part that changes with each item is still processed fresh.
How do I get the most from caching?
Keep everything that stays the same at the start of the prompt and the part that changes at the end, and send the whole job at once so similar items run together while the shared start is still in memory. The longer and more consistent that start, the bigger the saving.
Does Otium charge for cached input?
No. When a prompt starts with text that has just been processed, that part is served from cache and costs nothing. You pay for new input and for output. Sending your work all at once, rather than a few requests at a time, lets more of it share the cache.
Related guides
Tokens are what AI models read and write, and what you are billed on. What a token is, why output costs more than input, and how to estimate a job up front.
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
If your AI work is a pipeline, submitting one request at a time discards the structure that would let it be scheduled well. What the research says that costs.