AI inference, explained for the people who run it
Most AI services are built to answer in seconds, and price every request that way. A lot of real AI work has nobody waiting for it, and work that can wait is much cheaper to run. These guides explain what running AI costs, what it runs on, how to choose and shrink a model, how to run a multi-step pipeline, and what happens to your data.
Featured
Workflow-aware inference: why the request is the wrong unit
Why does it matter how you send a multi-step AI job? Most real AI work is a pipeline, but most AI services accept one request at a time. That throws away the information a scheduler needs to run the pipeline well, and three independent research groups have now measured what it costs.
Start here
What is batch AI inference?
Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.
5 min read →What an LLM inference request really costs
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
11 min read →How deadlines become discounts: SLA tiers explained
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
4 min read →Fundamentals
3 guidesThe words you need first: tokens, batch versus real-time, and why getting answers fast and getting a lot done cheaply pull against each other.
What is batch AI inference?
Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.
5 min readTokens, explained: how AI inference is priced
Tokens are what AI models read and write, and what you are billed on. What a token is, why output costs more than input, and how to estimate a job up front.
4 min readThroughput vs. latency: why batch wins on cost
Latency is how fast one request finishes; throughput is how much work you get per dollar. Why they pull against each other, and which one batch work should buy.
5 min readCost & economics
7 guidesWhere the money goes: GPU time, how much work a GPU gets through, caching, and how a deadline becomes a discount.
What an LLM inference request really costs
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
11 min readEstimating what an AI job will cost before you run it
From tokens to dollars for a whole pipeline: measuring token counts, adding retries, branches and per-call fees, and planning for the expensive case.
7 min readHow deadlines become discounts: SLA tiers explained
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
4 min readHow to choose a deadline for AI work
Work backwards from when the output is needed, not from how fast it could be. A practical way to set deadlines for batch AI jobs and pipelines.
5 min readPrompt caching and batch economics
Prompt caching reuses the computed state of a repeated prefix so you stop paying full price for it. How it works, why batch benefits most, and how it is billed.
5 min readHow to benchmark LLM inference honestly
Most published inference numbers are not comparable to anything. What a defensible benchmark records, why you measure a curve, and the mistakes that void one.
8 min readRenting vs owning a GPU for open-weight models
Should you buy a GPU to run open models or rent one? A way to compare that needs no card prices: hours of renting per $1,000, after electricity.
6 min readModels & quality
8 guidesOpen versus proprietary models, choosing the right size, and shrinking a model without hurting its answers.
Open-weight vs. proprietary models: when open is enough
Open-weight models are competitive on many batch tasks. The real trade-offs in cost, control, privacy and quality, and how to measure them on your workload.
9 min readHow to choose a model for a batch task
Bigger is not automatically better. How to match model size to a batch task, weigh quality against cost, and send only the hard items to a bigger model.
4 min readTemperature, top-p and max_tokens: the settings that shape an answer
What temperature, top-p, max_tokens, stop sequences, penalties and seeds do to a model’s answer, and sensible starting points for batch work.
6 min readMixture-of-Experts, explained: big-model quality at small-model cost
An MoE model holds many parameters in memory but activates a fraction per token. Total drives memory, active drives speed, and that split changes the economics.
6 min readMany drafts, one editor: the mixture-of-agents pattern
Several cheap drafts and one model that writes the answer from the best of them. What the research found, when it pays, what it costs, and how to build it.
10 min readQuantization explained: FP8, INT4, and serving models cheaper
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.
6 min readMeasuring inference quality: building a gold set
You cannot choose a model, quantization or prompt without measuring quality. How to build a small gold set and a test loop that turns choices into data.
4 min readGetting reliable structured output from LLMs
Schemas, validation, retries and escalation: how to get JSON from a language model that your code can trust, and what to do with the answers that fail.
6 min readInfrastructure
4 guidesThe machines and markets underneath: GPUs, spare (spot) capacity, how much one machine can serve at once, and what starting a machine costs.
Cold starts: the hidden cost of short inference jobs
Before a GPU produces a token it must boot, install a serving stack and load weights. On short jobs that setup can outcost the inference, and most is removable.
6 min readGPUs for AI inference: T4, L4, L40S, A100, H100
The GPU a model runs on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.
4 min readThe concurrency cliff: why inference throughput collapses instead of slowing down
Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.
9 min readSpot, preemptible, and remnant GPU compute
Spot and remnant GPUs are the same hardware sold cheap, with the catch that it can be reclaimed. How it works, and why batch can use it when interactive cannot.
4 min readIntegration
5 guidesGetting work in and out: the OpenAI-compatible batch API, the request file, and sending a whole pipeline as one workflow.
Migrating from the OpenAI Batch API
OpenAI-compatible batch surfaces make switching providers a base-URL change, not a rewrite. How the file-and-batch flow works, and what to check when you move.
4 min readStructuring a batch job: the JSONL request file
A batch job is a JSONL file, one request per line. How to structure it well: stable custom IDs, consistent prompts, and keeping large runs manageable.
5 min readFrom chained calls to one pipeline: designing multi-step AI work
Most useful AI work is several model calls in a row. The common patterns, where chained jobs lose time and money, and how to design steps that hold up.
7 min readPrompt, loop and graph engineering: three ways to get work out of a model
Prompt engineering shapes one call, loop engineering the control flow, graph engineering the whole job as a DAG. What each is, and when it is the right tool.
7 min readWorkflow-aware inference: why the request is the wrong unit
If your AI work is a pipeline, submitting one request at a time discards the structure that would let it be scheduled well. What the research says that costs.
10 min readPrivacy & security
7 guidesWhat happens to your data: how long it is kept, whether it is trained on, and who can see it. And what a model can reach: prompt injection, credentials, and the model files themselves.
Privacy in AI inference: retention, training, and isolation
Is your data retained, trained on, or readable by someone else? The privacy questions that matter in AI inference, and how open models change the answers.
5 min readData residency for AI inference: pick a boundary, not a region
Compliance obligations are jurisdictional, not regional. Why a residency boundary is the right primitive for AI inference, and how a lock must behave.
5 min readWho is actually running your inference?
Encryption does not protect a prompt during inference: the model needs plaintext in memory. That makes the machine, and whoever controls it, part of your risk.
6 min readPrompt injection in AI pipelines versus agents
An injected instruction can only do what the system around the model allows. Where the three ingredients of an attack live in an agent loop and a fixed graph.
9 min readWhere the secrets go: least privilege for AI tool calls
A model step should hold the authority its one task needs and no more, as a CI job does. Where credentials sit in an agent, a tool server and a fixed graph.
7 min readModel supply chain security: pickle, safetensors, pinned revisions and backdoors
A model file can run code when it loads, a repository can change under you, and weights can carry a trigger. What stops each attack, and what nothing stops yet.
8 min readA security review checklist for an AI inference vendor
The questions a security reviewer should ask any AI inference vendor, what a good answer looks like, and the three things no vendor can honestly promise.
7 min readThe case for batch
It is also the idea behind the name. Otium is Latin for productive leisure. Its opposite, negotium, means business and urgency. We named the company after the first one.
Most AI work is not interactive
Enriching a product catalog, sorting a backlog, summarizing documents overnight: none of it has a person waiting for the answer. It needs to be done by a deadline, not in a second, and paying the price of an instant answer for it is money spent on speed nobody uses.
A deadline is a discount
Computing time can’t be stored: a GPU that sits idle for an hour has lost that hour for good, so clouds sell spare capacity cheaply when demand is low. Work that must answer now can’t wait for it. Work with a deadline can, and every extra hour you allow is another hour in which cheap capacity can turn up.
The model you build on can be taken away
A model you use through one of the big AI companies is something you don’t control. It can be retired on their schedule, limited to a pricier plan, repriced, quietly replaced by a version that answers differently, or withdrawn altogether, sometimes by a regulator rather than the company. Each of those breaks the prompts you tuned and the tests you trusted. An open model is different: its files are already copied in thousands of places, so no announcement can make it disappear. Otium runs open models only, so no other company decides when your model goes away, and when we move a size to a newer model, we tell you first.
A pipeline is not a pile of unrelated requests
Most real AI work is a chain of steps: pull out the facts, classify them, look something up, write a summary. But the services that run it take one request at a time and forget it, so they never learn that the steps belong together. Independent researchers have measured what that costs, and it is not small: the same instructions are processed again for every item, models are reloaded between steps, a badly formed answer costs a whole extra round, and steps that could run side by side wait in line. Send the whole pipeline as one workflow and Otium sees its whole shape: it keeps a model loaded across the steps that use it, retries a bad answer in place, and runs independent steps side by side.
Batching flattens the demand curve
Power grids and clouds are both sized for their busiest moment, not their average. Work that must answer instantly arrives in spikes, so machines are kept running in case a spike comes. Work that can wait can move into the quiet hours, overnight or onto capacity that is already idle, so fewer machines sit running just in case.
Flatter peaks mean less wasted power
The busiest hours on the grid are met by the dirtiest and most expensive power plants, the ones switched on only for peaks, and data centers increasingly cause those peaks. Moving work that can wait out of the peak means less of it runs on that power, and more on cheaper, often cleaner, power from the quiet hours. Using capacity that would otherwise sit idle wastes less money and less energy.
Flattening the demand curve
Machines are bought for the busiest moment, not the average one. The same work spread into the quiet hours needs far fewer machines kept running, and avoids the dirtiest power, which is switched on only for the peaks.
An illustration, not a measurement. The point is the mechanism: work that can wait moves off the peak and onto capacity that would otherwise sit idle. The measurement is public too: spot prices show the troughs, and we publish them as live GPU prices and a monthly market report.
Send us the work that can wait.
Work that can wait runs on cheaper capacity and on open models nobody can retire. Leave your email and we’ll send you a link to create your account.
Free to try on our demo model. No credit card, no spam: we email you a link to finish.
Otium is in beta and open to everyone. Already have an account? Sign in.