AI inference, explained for the people who run it
The commercial inference market optimizes for latency. A large share of real AI work has no human waiting on a spinner, and treating it as deferrable makes it much cheaper. These guides cover how inference is priced, what runs it, how to pick and shrink models, and what happens to your data.
Featured
Workflow-aware inference: why the request is the wrong unit
Most production AI work is a pipeline, but most inference APIs accept one request at a time. That mismatch throws away the structure a scheduler needs to do anything clever with the work — and three independent research groups have now measured what it costs.
Start here
What is batch AI inference?
Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.
7 min read →What an LLM inference request really costs
Behind every per-token price is a GPU running for some number of seconds. The GPU-seconds model of inference cost, and the two levers that actually move it.
9 min read →How deadlines become discounts: SLA tiers explained
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
7 min read →Fundamentals
3 guidesThe vocabulary of AI inference — tokens, batch vs. real-time, and the throughput-for-latency trade underneath both.
What is batch AI inference?
Batch AI inference runs large volumes of requests against a deadline instead of in real time. How it differs from interactive inference, and why it costs less.
7 min readTokens, explained: how AI inference is priced
Tokens are what AI models read and write, and what you are billed on. What a token is, why output costs more than input, and how to estimate a job up front.
8 min readThroughput vs. latency: why batch wins on cost
Latency is how fast one request finishes; throughput is how much work you get per dollar. Why they pull against each other, and which one batch work should buy.
7 min readCost & economics
4 guidesWhere the money goes: GPU-seconds, throughput, caching, and how a deadline becomes a discount.
What an LLM inference request really costs
Behind every per-token price is a GPU running for some number of seconds. The GPU-seconds model of inference cost, and the two levers that actually move it.
9 min readHow deadlines become discounts: SLA tiers explained
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
7 min readPrompt caching and batch economics
Prompt caching reuses the computed state of a repeated prefix so you stop paying full price for it. How it works, why batch benefits most, and how it is billed.
8 min readHow to benchmark LLM inference honestly
Most published inference numbers are not comparable to anything. What a defensible benchmark records, why you measure a curve, and the mistakes that void one.
9 min readModels & quality
5 guidesOpen vs. proprietary, choosing the right size, and quantization without wrecking quality.
Open-weight vs. proprietary models: when open is enough
Open-weight models have closed much of the quality gap with proprietary APIs. The real trade-offs in cost, control, privacy and quality for batch workloads.
8 min readHow to choose a model for a batch task
Bigger is not automatically better. A practical method for matching model size and family to a batch task, balancing quality against cost and throughput.
8 min readMixture-of-Experts, explained: big-model quality at small-model cost
An MoE model holds many parameters in memory but activates a fraction per token. Total drives memory, active drives speed, and that split changes the economics.
9 min readQuantization explained: FP8, INT4, and serving models cheaper
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.
9 min readMeasuring inference quality: building a gold set
You cannot choose a model, quantization or prompt without measuring quality. How to build a small gold set and an eval harness that turns choices into data.
8 min readInfrastructure
4 guidesThe hardware and markets underneath inference — GPUs, spot capacity, concurrency ceilings, and what a cold start costs.
Cold starts: the hidden cost of short inference jobs
Before a GPU produces a token it must boot, install a serving stack and load weights. On short jobs that setup can outcost the inference, and most is removable.
8 min readGPUs for AI inference: T4, L4, L40S, A100, H100
The GPU you serve a model on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.
9 min readThe concurrency cliff: why inference throughput collapses instead of slowing down
Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.
8 min readSpot, preemptible, and remnant GPU compute
Spot and remnant GPUs are the same hardware sold cheap, with the catch that it can be reclaimed. How it works, and why batch can use it when interactive cannot.
8 min readIntegration
3 guidesGetting work in and out: the OpenAI-compatible batch surface, the JSONL request file, and describing a whole pipeline at once.
Migrating from the OpenAI Batch API
OpenAI-compatible batch surfaces make switching providers a base-URL change, not a rewrite. How the file-and-batch flow works, and what to check when you move.
7 min readStructuring a batch job: the JSONL request file
A batch job is a JSONL file, one request per line. How to structure it well: stable custom IDs, prompts designed for caching, and keeping large runs manageable.
7 min readWorkflow-aware inference: why the request is the wrong unit
If your AI work is a pipeline, submitting one request at a time discards the structure that would let it be scheduled well. What the research says that costs.
11 min readPrivacy & trust
3 guidesRetention, training, and isolation — what actually happens to your data during inference.
Privacy in AI inference: retention, training, and isolation
Is your data retained, trained on, or readable by someone else? The privacy questions that matter in AI inference, and how open models change the answers.
8 min readData residency for AI inference: pick a boundary, not a region
Compliance obligations are jurisdictional, not regional. Why a residency boundary is the right primitive for AI inference, and how a lock must behave.
8 min readWho is actually running your inference?
Encryption does not protect a prompt during inference: the model needs plaintext in memory. That makes the machine, and whoever controls it, part of your risk.
8 min readThe case for batch
It is also the idea behind the name. Otium is Latin for productive leisure — its opposite, negotium, means business and urgency. We named the company after the first one.
Most AI work is not interactive
Enriching a catalog, classifying a backlog, summarizing documents overnight, regenerating embeddings after a schema change — none of it has a human watching a spinner. It needs to be done by a deadline, not in a millisecond. Paying interactive prices for it is pure waste.
A deadline is a discount
Compute is perishable: an idle GPU-second is gone forever, so it gets sold off cheap when demand is low. Interactive work can’t use it — it needs capacity hot and waiting at 2pm. Batch work can. Every hour of slack you give us is another hour in which cheap compute might show up.
The model you build on can be taken away
A hosted frontier model is a dependency you don’t control. It can be deprecated on the vendor’s schedule, restricted to new tiers, priced differently, silently swapped for a newer version that answers differently — or pulled from general availability entirely by someone who isn’t even the vendor. Every one of those events invalidates the prompts you tuned and the evals you trusted. Open weights change that: the file already sits on disk in thousands of places, so no announcement makes it stop existing. Otium runs open models only, which is why we can commit to something a hosted vendor cannot — that the model you built against will still be there.
A pipeline is not a pile of unrelated requests
Most production AI work is a chain — extract, classify, look something up, summarize — but the APIs serving it take one request at a time and forget it. That throws away the one fact worth the most: that the steps belong together. Independent research groups have now measured what that costs, and the answer is not small; the same instructions get re-processed for every item, models get reloaded between steps, a malformed answer costs a whole extra round trip, and steps that could have run side by side run in sequence because nothing knew they were unrelated. Describe the whole pipeline up front and a scheduler can do something about all four.
Batching flattens the demand curve
The grid and the cloud both struggle with peaks, not totals. Interactive demand is spiky and must be met instantly with always-hot, over-provisioned hardware. Deferrable work can be shifted into the troughs — overnight, off-peak, into idle capacity that already exists — so less hardware sits hot waiting for a spike.
Flatter peaks mean less wasted power
Electricity demand peaks are met by the dirtiest, most expensive “peaker” plants, and data centers increasingly drive those peaks. Shifting deferrable compute off the peak reduces the marginal load that peakers serve and lets work soak up cheaper, often cleaner, off-peak and surplus energy. Using capacity that would otherwise sit idle wastes less of both money and power.
Flattening the demand curve
Peaks, not totals, are what force over-provisioning. The same work spread into the troughs needs far less hardware kept hot, and avoids the dirtiest power brought online to meet a spike.
Illustrative, not measured. The mechanism is the point: deferrable load moves off the peak and onto capacity that would otherwise be idle.
Send us the work that can wait.
Deferrable work runs on cheaper capacity and open models nobody can retire. Leave your email and we’ll invite you to the alpha.
No credit card, no spam — one email when your invite is ready.
Closed alpha — onboarding is gated while we calibrate. Already invited? Sign in.