Cold starts: the hidden cost of short inference jobs
A GPU-hour price is easy to compare across providers. What it never tells you is how much of that hour passes before the model produces anything at all. That setup time, the cold start, is paid every time a machine starts, and on short jobs it can cost more than the work itself.
What a cold start actually consists of
Between deciding to run a job and getting the first word of output, a fresh GPU machine has to go through a fixed sequence. None of it produces output, and all of it is billed.
- Start the machine: the cloud finds hardware and the operating system boots.
- Get the serving software in place: the program that runs the model, and its large set of dependencies.
- Fetch the model’s weights (the model itself, often many gigabytes) from wherever they are stored.
- Load the weights into the GPU’s memory and get the software ready.
- Produce the first token (roughly, the first word).
Most people assume step 4 is the slow part, because loading a model of many gigabytes sounds expensive. It usually is not.
Where the time actually goes
In our own early runs, two steps took most of the wait, and neither needed to happen at all: installing serving software that could have been installed in advance, and downloading the weights from a public model hub across the internet when they could have been stored next to the machine. Loading the weights into the GPU and producing the first token were a small part.
Both fixes are well known, and both are easy to lose. A machine image built before the "already installed" check was added will quietly install everything again on every start. A fix you believe you shipped is worth measuring.
When it matters, and when it does not
A cold start is a fixed cost, so what matters is how much useful work it is spread over. The arithmetic is simple and worth knowing:
| Useful work on the machine | Overhead with a 10-minute cold start | Overhead with a 90-second cold start |
|---|---|---|
| 5 minutes | 67% | 23% |
| 30 minutes | 25% | 5% |
| 4 hours | 4% | 0.6% |
A five-minute job on a slow-starting machine spends most of its money on setup. The same machine running for four hours barely notices. This is the strongest argument for gathering work before starting a machine, and the reason a batch platform that fills a machine is cheaper by design than one that starts a machine for each job.
How to cut it
- Pre-install the software. Build the serving software, drivers and runtime into the machine image, so starting a machine is not an installation.
- Store weights next to the machines. Downloading from a public hub across the internet is slow; reading from storage in the same region is many times faster.
- Check for the pre-installed software explicitly. Mark the image as ready rather than probing at start-up: a probe that behaves differently in some environments will silently reinstall everything.
- Keep machines busy while there is work. A machine that has already paid the setup cost should keep taking work rather than being shut down and rebuilt.
- Run bigger batches. The most effective fix is not a faster cold start but more work to spread it over.
- Avoid switching models mid-run. Changing models reloads weights: a cold start you pay without even restarting the machine.
Why this favours batch
Real-time AI services cannot spread a cold start over anything, because a person is waiting. That is why they keep hardware switched on and idle, and why you pay for that readiness in every token whether you needed it or not.
Batch work is the opposite. A deadline gives the scheduler room to gather enough work that the setup cost becomes small, to reuse a machine that has already paid it, and to prefer machines where the weights are already stored. None of that is possible for a system that must answer at once, and it is a large part of why the same model costs very different amounts depending on how it is served.
On Otium, this is why your deadline is not a start time: work may be held to gather it onto one machine. With nothing already running, allow up to an hour for the first step to start: finding capacity, starting a machine and loading the model.
It is also why the best thing you can do about cold starts is send all your work at once. Starting a machine costs the same for ten requests or ten thousand, and Otium waits until there is enough work to fill one before starting it. A whole backlog fills it straight away and shares the start-up across every request; the same work sent a few requests at a time waits to be gathered, and each start is spread over less.
Sources
The finding above, that fetching the weights dominates start-up and that where they are stored matters more than how big they are, is the subject of a systems paper:
By harnessing the substantial near-GPU storage and memory capacities of inference servers, ServerlessLLM achieves effective local checkpoint storage, minimizing the need for remote checkpoint downloads and ensuring efficient checkpoint loading.
They report waits 10 to 200 times shorter across workloads. That headline number does not carry over, because they compare against a cold serverless call, not a batch machine. What carries over is the diagnosis: the expensive part is fetching weights over a network that is not local, and the fix is storage next to the machine rather than a faster download.
Frequently asked questions
Is cold start included in the price I am quoted?
It depends on the provider, so ask. If you pay per token, it is built into the rate; if you rent machines, you pay for it as time on the clock. Either way someone pays, so a provider that has not reduced it is either losing margin or passing the cost to you.
How fast can a cold start realistically get?
With the software pre-installed and the weights stored in the same region, a minute or two to the first token is a reasonable target for a mid-sized model. The floor is the time to get a machine plus the time to load the weights into the GPU’s memory, which cannot be avoided.
How do I keep cold starts from showing up in my bill?
Send your work in big batches rather than a few requests at a time. A start-up costs the same however much work follows it, so the more work arrives together, the smaller its share of each request. On Otium, a whole backlog sent at once also starts sooner, because there is enough of it to fill a machine.
Does a smaller model always start faster?
Usually, because there is less to move and load, but only if the serving software is already in place. If most of your cold start is installing software, a smaller model barely helps.
Related guides
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
Spot and remnant GPUs are the same hardware sold cheap, with the catch that it can be reclaimed. How it works, and why batch can use it when interactive cannot.
Latency is how fast one request finishes; throughput is how much work you get per dollar. Why they pull against each other, and which one batch work should buy.
Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.