Spot, preemptible, and remnant GPU compute
Why are some cloud GPUs so much cheaper than others? Because the cloud can take them back at short notice. For a chat app that waits on every answer, that is a dealbreaker. For batch work with a deadline, it is the cheapest compute there is.
What spot and remnant capacity are
Clouds build enough machines for their busiest hours, so at any moment some of them sit unused. Rather than leave them idle, the cloud sells that spare capacity at a discount. AWS calls it spot instances, Google Cloud calls it preemptible or spot VMs, and Azure calls it spot or low-priority VMs. "Remnant" is the general term: leftover capacity, sold cheap.
The catch is in the terms. The cloud can take the machine back, with little or no warning, when a full-price customer needs it. In return, the same GPU often costs 50% to 90% less than the standard (on-demand) price.
Why batch work can use it and interactive cannot
An interactive service cannot have its machine taken away in the middle of an answer: the person waiting sees an error. So interactive systems mostly pay for capacity that cannot be taken back.
Batch work only has to be finished by a deadline. If a machine is taken back, the work it was doing is not lost; it goes back in the queue and runs on another machine. The interruption costs a little time, and the result is the same.
The price changes by place and by hour
There is no single spot price. It differs by machine type and by region, and it moves through the day as demand rises and falls. The same GPU can cost much less in one region than another at the same moment, and less at 3am than at midday.
A deadline is what lets you use those differences. With time to wait, a scheduler can look across regions and clouds and wait for a cheap moment, instead of taking the first price it sees. The longer the deadline, the more of the market it can look at and the lower the price it can wait for.
The deadline gives the scheduler time; your work gives it something to put on the cheap machine once it finds one. Starting a machine costs the same for ten requests or ten thousand, so send the whole backlog at once and a cheap machine starts full, rather than waiting for a trickle of requests to be gathered.
You can see how far prices spread: Otium publishes a monthly GPU price report, and a table of today’s spot and on-demand prices for every GPU machine in every region we track.
Managing the downside
Waiting for a cheap moment has a risk: the cheap moment may not come before the deadline, and a region can run short of a particular GPU. A careful batch platform plans for both. It prices from a price that has held for a while, not a brief low, and it keeps standard on-demand capacity as a backstop: if the deadline gets close and cheap capacity has not appeared, it pays the higher price rather than miss the deadline.
Knowing when to stop waiting is a well-studied problem. The paper Can't Be Late (NSDI 2024, which won an Outstanding Paper award) works through exactly this case: use spot by preference, keep on-demand in reserve, and switch depending on whether the job is ahead of or behind the progress its deadline needs. It reports spot at 60% to 90% less than on-demand, in line with the range above.
The other half of the idea, looking across regions rather than waiting in one, has been measured too:
Spot instances offer 3-10x lower cost than on-demand instances, but their unpredictable availability makes meeting deadlines difficult.
In real cloud deployments, SkyNomad reports that the alternatives it compared against cost 1.25 to 3.96 times as much as using spot across many regions. Both papers assume one thing: a deadline. Without one there is nothing to trade, and you are back to paying on-demand prices.
Sources
Frequently asked questions
How much cheaper is spot GPU capacity?
It depends on the GPU, the region and the time, but 50% to 90% less than on-demand for the identical hardware is common. The catch is that the cloud can take the machine back with little warning.
What happens to my job if a spot instance is reclaimed?
On a well-built batch platform, the unfinished work goes back in the queue and runs on another machine. Because progress is recorded outside the machine and running a job twice is safe, an interruption costs some time, not your results.
Related guides
Before a GPU produces a token it must boot, install a serving stack and load weights. On short jobs that setup can outcost the inference, and most is removable.
A deadline is the most valuable thing you can give a batch inference platform. How SLA tiers turn patience into price, and how to pick the right window.
The GPU a model runs on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
Should you buy a GPU to run open models or rent one? A way to compare that needs no card prices: hours of renting per $1,000, after electricity.