Cold starts: the hidden cost of short inference jobs
A GPU-hour price is easy to compare across providers. What that number never tells you is how much of the hour goes by before the model produces anything at all.
What a cold start actually consists of
Between deciding to run a job and receiving the first token, a fresh GPU instance has to work through a fixed sequence. None of it produces output, and all of it is billed.
- Provision and boot the instance — the cloud allocates hardware and the OS comes up.
- Get a serving stack in place — the inference server and its (large) dependency tree.
- Fetch the model weights from wherever they live.
- Load those weights into VRAM and warm the runtime.
- Produce the first token.
People tend to assume step 4 dominates, because loading a multi-gigabyte model sounds like the expensive part. In our measurements it was not close.
A measured breakdown
From one of our own bring-up runs on a 24 GB spot instance, timed end to end from boot to first token:
| Phase | Seconds | Share | Removable by |
|---|---|---|---|
| VM boot, including installing the serving stack (~215 s) | 275 | 50% | Baking the stack into the machine image |
| Weight load, dominated by downloading from a public model hub | 272 | 49% | A weight cache in the same region as the compute |
| First token | 4 | 1% | Nothing — this is the real work |
| Total | 551 | 100% |
Nine minutes to first token, of which roughly eight were spent on two tasks that produce nothing and did not need to happen at all: installing software that could have been pre-installed, and downloading weights across the public internet that could have been sitting next door.
When it matters, and when it does not
Cold start is a fixed cost, so its significance depends entirely on how much useful work you amortize it over. The arithmetic is simple and worth internalising:
| Useful inference time | Overhead at 551 s cold start | Overhead at 90 s cold start |
|---|---|---|
| 5 minutes | 65% | 23% |
| 30 minutes | 23% | 5% |
| 4 hours | 4% | 0.6% |
A five-minute job on an unoptimised worker spends most of its money on setup. The same worker running for four hours barely notices. This is the single strongest argument for accumulating work before you provision anything, and the reason a batch platform that fills a machine is structurally cheaper than one that spins up per job.
How to cut it
- Bake the image. Pre-install the serving stack, drivers, and runtime into the machine image so boot does not become an install. This was half our cold start.
- Cache weights next to the compute. Pulling from a public hub across the internet is the other half; pulling from storage in the same region is roughly an order of magnitude faster.
- Trust your own marker. Detect the pre-baked stack with an explicit marker rather than probing at boot — a probe that behaves differently under a non-login shell will silently reinstall everything.
- Keep workers warm while the queue is deep. A machine that has already paid the tax should keep taking work rather than being torn down and rebuilt.
- Batch bigger. The most effective lever is not making cold start faster, it is having more work to spread it over.
- Pick models that fit. Swapping models mid-run reloads weights, which is a cold start you pay without rebooting anything.
Why this favours batch
Interactive inference cannot amortize a cold start, because a user is waiting — which is precisely why interactive providers keep hardware hot and idle, and why you pay for that readiness in every token whether you needed it or not.
Batch work has the opposite shape. A deadline gives a scheduler room to accumulate enough work that the setup cost becomes a rounding error, to reuse a machine that has already paid it, and to prefer capacity where the weights are already staged. None of that is available to a system that has to answer immediately, and it is a large part of why the same model costs radically different amounts depending on how it is served.
Sources
Our breakdown above is one run on one instance type. The finding it points at — that checkpoint loading dominates startup, and that where the weights live matters more than how big they are — is the explicit subject of a systems paper:
By harnessing the substantial near-GPU storage and memory capacities of inference servers, ServerlessLLM achieves effective local checkpoint storage, minimizing the need for remote checkpoint downloads and ensuring efficient checkpoint loading.
They report latency reductions of 10-200x across workloads. The headline number is not the transferable part — their baseline is a cold serverless invocation, not a batch worker. The transferable part is the diagnosis, which matches ours exactly: the expensive thing is fetching weights over a network that is not local, and the fix is storage that sits next to the compute rather than a faster download.
Frequently asked questions
Is cold start included in the price I am quoted?
It depends on the provider, and it is worth asking directly. If you are billed per token it is absorbed into the rate; if you are renting instances it is billed to you as wall-clock time. Either way somebody pays for it, so a provider who has not optimised it is either eating margin or passing it on.
How fast can a cold start realistically get?
With the serving stack pre-baked and weights staged regionally, first token in roughly a minute to a minute and a half is a reasonable target for a model of this size. The floor is instance provisioning plus the unavoidable load of weights into VRAM.
Does a smaller model always start faster?
Usually, because there is less to move and load — but only if the serving stack is already in place. If your cold start is dominated by installing software, shrinking the model barely helps.
Related guides
Behind every per-token price is a GPU running for some number of seconds. The GPU-seconds model of inference cost, and the two levers that actually move it.
Spot and remnant GPUs are the same hardware sold cheap, with the catch that it can be reclaimed. How it works, and why batch can use it when interactive cannot.
Latency is how fast one request finishes; throughput is how much work you get per dollar. Why they pull against each other, and which one batch work should buy.
Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.