Spot, preemptible, and remnant GPU compute
The cheapest GPUs in any cloud are the ones the provider might take back at a moment notice. For interactive systems that is a dealbreaker. For batch work it is an opportunity.
What spot and remnant capacity are
Cloud providers build out enormous fleets to meet peak demand, which means at any given moment some hardware is unused. Rather than let it sit idle, they sell that slack at a steep discount under names like spot instances (AWS), preemptible VMs (GCP), or low-priority VMs (Azure). "Remnant" is the general term: leftover capacity offered cheap.
The catch is in the contract. The provider can reclaim the instance with little or no notice when it needs the capacity back for a full-price customer. In exchange, the price can be a fraction of on-demand — often 50-90% cheaper for the exact same GPU.
Why batch work can use it and interactive cannot
An interactive service cannot tolerate having its hardware yanked mid-request; a reclaim means dropped sessions and broken latency guarantees. So interactive systems mostly stay on expensive, guaranteed capacity.
Batch work has a different relationship with interruption. Because results are only due by a deadline, a reclaimed instance is not a failure — it is just a pause. The in-flight work re-queues and resumes elsewhere. As long as the platform tracks progress durably and is idempotent about retries, interruptions cost a little time, not correctness.
Prices vary across a market
Spot prices are not one number. They vary by instance type, by region, by availability zone, and over time as supply and demand shift. The same GPU can be much cheaper in one region than another at the same moment, and cheaper at 3am than at peak.
This variation is precisely what a deadline lets you exploit. With time to wait, a scheduler can shop across regions and providers and hold out for a trough instead of taking the first available price. The wider the deadline, the more of the market it can survey and the lower the price it can wait for. That is the arbitrage deadline-flexible inference runs on.
Managing the downside
Riding spot is not free of risk: a trough you are waiting for might not arrive before a deadline, and a region can run short of a particular GPU. A robust batch platform manages this with conservative quoting (pricing against a sustained price level, not a fleeting low) and an on-demand fallback as a stop-loss — if a deadline approaches and cheap capacity has not appeared, it pays up rather than miss the SLA.
That "knowing when to stop waiting" is a real scheduling problem with a literature behind it. The NSDI 2024 paper Can't Be Late formalises exactly this shape — spot by preference, on-demand held in reserve, switching governed by whether the job is running ahead of or behind the progress its deadline requires — and won an Outstanding Paper award for it. Its reported figure for how much cheaper spot is, 60% to 90% against on-demand, sits inside the range quoted above.
The other half of the arbitrage — shopping across regions rather than waiting in one — has been measured too:
Spot instances offer 3-10x lower cost than on-demand instances, but their unpredictable availability makes meeting deadlines difficult.
SkyNomad reports 1.25-3.96x cost savings in real cloud deployments from treating regional heterogeneity as the resource to exploit. Note what both papers assume: a deadline. Without one there is nothing to trade, and the whole strategy collapses back to paying on-demand prices.
Sources
Frequently asked questions
How much cheaper is spot GPU capacity?
It varies by GPU, region, and time, but discounts of 50-90% versus on-demand for the identical hardware are common. The trade is that the instance can be reclaimed with little notice.
What happens to my job if a spot instance is reclaimed?
On a well-built batch platform, the in-flight work re-queues and resumes on other capacity. Because job state is durable and execution is idempotent, an interruption costs a little time, not your results.
Related guides
Before a GPU produces a token it must boot, install a serving stack and load weights. On short jobs that setup can outcost the inference, and most is removable.
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
The GPU you serve a model on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.
Behind every per-token price is a GPU running for some number of seconds. The GPU-seconds model of inference cost, and the two levers that actually move it.