What is batch AI inference?
Most writing about AI models assumes a person is waiting for each answer. A large share of real AI work has nobody waiting: it only has to be done by a deadline. That work is batch work, and it can run for a fraction of the price.
Two shapes of inference
Inference means running a trained model to get an output: a reply, a label, a summary, an embedding. It comes in two shapes, and they cost very different amounts to run.
Interactive (or real-time) inference happens while someone waits: a chat reply, an autocomplete, a search box. The answer loses its value within seconds, so the provider has to keep machines switched on and ready for the moment a request arrives.
Batch inference happens against a deadline. You have ten thousand product descriptions to write, a million rows to label, a backlog of documents to summarize overnight. Nobody watches a spinner. What matters is that the whole job is done, and done right, by a time you choose.
Why batch costs less
Most of the price of interactive inference pays for being ready, not for the work. To answer instantly, a provider keeps more machines running than it needs, so there is always room for a sudden rush. That idle capacity is built into the price of every request, whether or not you needed an answer in under a second.
Batch work does not need that readiness. Because a result is fine any time before the deadline, the system can:
- Run many requests at once on each GPU, keeping it close to fully busy instead of idle between requests.
- Wait for cheap capacity: spare cloud GPUs sold at a discount (spot capacity) instead of guaranteed, always-on machines.
- Shrug off interruptions: if a cheap machine is taken back mid-job, the unfinished work goes back in the queue and finishes on another one.
These savings multiply. A busy GPU lowers the cost of each request; a cheap GPU lowers the cost of each hour. Together they are why a batch service can cost a fraction of the same model served interactively.
What kinds of work are batch work
Once you look for it, work that can wait is everywhere:
- Enriching a catalog: writing descriptions, tags, titles or attributes for every product.
- Labelling, routing or scoring a backlog of tickets, reviews or records.
- Summarizing or pulling facts out of contracts, tickets, transcripts or research.
- Rebuilding embeddings after you change a model or a schema.
- Generating test data and running offline evaluations.
- Nightly or weekly reprocessing that feeds dashboards and search.
None of these needs an answer this second. Many still run on interactive endpoints, because that is the default an API offers.
A lot of this work also comes in steps: pull out the facts, then classify them, then write something from both. Sent as separate batches, each step waits for the one before it, and the waits add up. Sent together as one workflow, the whole pipeline shares a single deadline.
How a batch job runs
A batch service separates accepting your work from running it. You send a file of requests with a deadline; the service stores it safely and replies straight away. Then a scheduler decides when and where to run it so it costs least within your window. It starts machines, runs your requests through the model while keeping each GPU busy, writes back the results and tells you the job is done.
That separation is what moves the price. The longer the deadline you can give, the more freedom the scheduler has to find a cheap moment. That trade is what Otium is built around.
It also means you can send everything at once. A batch service is a queue you do not have to run: no rate limiter, no retry loop, no feeding a backlog through a few requests at a time. And the more work it can see together, the better it does. On Otium, a whole backlog fills a machine straight away instead of waiting to be gathered, the cost of starting that machine is shared across all of it, and requests that share instructions are served from cache, where input is free.
Where this is established
None of this is folklore. Batching requests together has a decade of serving research behind it, using cheap, interruptible capacity against a deadline has been studied as a scheduling problem, and the large interactive providers sell a batch option of their own.
That last one is a useful reference point. OpenAI's Batch API offers a 50% discount over its synchronous prices, and its completion window can only be set to 24 hours. One fixed window, one flat discount: the trade at its simplest. A service that lets you choose the deadline can go further.
Frequently asked questions
Is batch inference slower or lower quality?
Not lower quality: it runs the same models and produces the same outputs. Only the timing changes. The work runs within your deadline rather than immediately, which is what makes it cheaper.
Should I send my work all at once or a bit at a time?
All at once. A batch service holds the work, retries what fails and delivers by the deadline, so there is nothing to gain by pacing it. Big batches fill machines sooner and share their start-up cost, and on Otium requests that share instructions are served from cache for free.
How long can a batch job take?
You decide, by setting a deadline. Longer deadlines (hours to days) reach the cheapest capacity; shorter ones cost more, because they leave less time to wait for a good price.
Related guides
Latency is how fast one request finishes; throughput is how much work you get per dollar. Why they pull against each other, and which one batch work should buy.
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
OpenAI-compatible batch surfaces make switching providers a base-URL change, not a rewrite. How the file-and-batch flow works, and what to check when you move.