Prompt caching and batch economics

What prompt caching is

When a model reads a prompt, it computes an internal representation (the attention key/value state) for every token. Prompt caching saves that computed state for a prefix so that the next request sharing the same prefix can skip recomputing it and reuse the cached state instead.

The savings apply to the shared prefix only — typically your system prompt, instructions, few-shot examples, and any fixed context. The part that varies per request (the actual item being processed) is still computed fresh.

Why batch workloads benefit most

Batch jobs are the ideal case for caching because they are highly repetitive by construction. Enriching a product catalog means sending the same elaborate instructions with each of thousands of items. The fixed prefix might be hundreds or thousands of tokens; the variable part might be small. Cache the prefix once and every subsequent item processes its shared context for a fraction of the cost.

How caching shows up in pricing

Providers expose caching as a third rate. Instead of one input price, a request is billed across three buckets:

Uncached input

Prefix tokens processed for the first time, or after the cache expired. Full input rate.

Cached input

Prefix tokens served from cache. Heavily discounted — often a small fraction of the uncached rate.

Output

Generated tokens. Unaffected by input caching; priced as usual.

bill = uncached_input x r_uncached
     + cached_input   x r_cached     (r_cached << r_uncached)
     + output         x r_output

A workload with a high cache-hit rate can see its effective input cost fall sharply, because most of the input is billed at the low cached rate. The important subtlety: the provider decides what counts as cached, based on what is actually in the cache at request time — so your realized savings depend on hit rates you do not fully control.

Guaranteed caching vs. best-effort

On interactive APIs, caching is typically best-effort: a cache entry lives for a short window and may be evicted under load, so two identical requests minutes apart can be billed very differently. Batch is friendlier territory. Because the platform controls scheduling, it can deliberately group items that share a prefix and keep that prefix warm for the whole job — turning best-effort caching into something much closer to a guarantee, and passing the structural savings through transparently.

The mechanism underneath has a name and a paper. SGLang introduced RadixAttention, which keeps completed requests' KV cache in a radix tree so a later request sharing a prefix finds it automatically rather than by being told:

The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding.

They report up to 6.4x higher throughput against contemporary systems. The word to notice is automatically: reuse happens when a shared prefix is still resident, which is a scheduling property. Two requests that share a 4,000-token preamble but are run an hour apart share nothing. Ordering the work so that they run together is worth more than any prompt-engineering trick, and it is not something the requester can do from outside.

Sources

Caching is the clearest case where knowing what is coming next is worth money — which is why it also shows up in the research on treating a whole workload, rather than a request, as the unit of optimization.