What is batch AI inference?

Two shapes of inference

Inference is the act of running a trained model to produce an output: a completion, a classification, an embedding, a summary. There are two fundamentally different shapes it takes, and they have almost nothing in common economically.

Interactive (or real-time) inference happens while someone waits. A chat reply, an autocomplete, a search box — the value of the answer decays in seconds, so the system must keep hardware hot and idle, ready to respond the instant a request arrives.

Batch inference happens against a deadline. You have ten thousand product descriptions to enrich, a million rows to classify, a backlog of documents to summarize overnight. Nobody is watching a spinner. The only thing that matters is that the whole job is done correctly by some time in the future.

Why batch is structurally cheaper

The price of interactive inference is dominated by readiness, not computation. To answer instantly, a provider must over-provision: keep GPUs powered, warm, and underutilized so there is always spare headroom for a spike. You pay for that standing capacity in every token, whether or not you needed sub-second latency.

Batch work removes the readiness requirement. Because a result is acceptable any time before the deadline, the system can:

• Pack many requests together to keep each GPU near 100% utilization instead of idling between interactive requests.
• Wait for cheap capacity — spot or remnant GPUs that come and go — instead of paying for guaranteed, always-on hardware.
• Tolerate interruption: if a cheap instance is reclaimed mid-job, the remaining work simply re-queues and finishes elsewhere.

Each of those levers compounds. High utilization lowers the cost per token. Cheap, interruptible capacity lowers the cost per GPU-second. Together they are why a batch lane can be many times cheaper than the same model served interactively.

What kinds of work are batch work

Once you start looking, deferrable inference is everywhere in a production system:

• Catalog and content enrichment — generating descriptions, tags, titles, or attributes across a product database.
• Classification and moderation of a backlog — labeling, routing, or scoring large datasets.
• Summarization and extraction over document corpora — contracts, tickets, transcripts, research.
• Embedding (re)generation after a model or schema change — rebuilding a vector index.
• Synthetic data generation and offline evaluation runs.
• Nightly or periodic reprocessing pipelines that feed dashboards and search.

None of these need an answer this second. All of them are frequently run on interactive endpoints simply because that is the default an API exposes.

How a batch job actually runs

A batch inference platform decouples acceptance from execution. You submit a file of requests with a deadline; the system stores it durably and returns immediately. Behind the scenes a scheduler decides when and where to run the work to minimize cost within your window, provisions compute, streams your requests through the model at high utilization, writes the results back, and tells you the job is done.

That decoupling is what makes the price move. The wider the deadline you can give, the more freedom the scheduler has to find a cheap moment to run — which is exactly the trade Otium is built around.

Where this is established

The three levers above are not folk wisdom; each has a body of measurement behind it. Batching is the subject of a decade of serving-systems research, interruptible capacity has been formalised as a scheduling problem with deadlines, and the major interactive vendors publish a batch lane of their own that makes the trade explicit.

That last one is a useful reference point. OpenAI's Batch API offers a 50% discount over its synchronous prices — and its completion window can only be set to 24 hours. One fixed window, one flat discount. That is the shape of the trade at its simplest, and it is the thing a deadline-flexible platform is trying to generalise.