Twelve minutes to boot a GPU worker, and where they went
Every worker Otium starts is a GPU that is billed from the moment it launches and serves nothing until the model is loaded. For our smallest model that took about twelve minutes, and I assumed that the time went on moving the weights. It did not. This is what the timings showed once we recorded them, and the five changes that brought the same boot to under eight minutes.
Why a boot is a cost line
Otium runs batch work on spot GPUs, machines a cloud rents cheaply because nobody else wants them right now. Nothing runs between batches, so most batches start with a cold boot: launch a machine, fetch the model, start the inference engine (vLLM), wait for it to report ready. The cloud bills every second of that, and not one token comes out of it.
When we started measuring, the average boot in production took about 14 minutes. A batch job pays that before its first token, every time a machine starts, and a cheaper GPU does not change it: the minutes are the same, only the rate per minute is lower. The only fixes are a shorter boot, or fewer of them.
Two weeks of wrong numbers
Each worker reports its own boot when it starts serving. For the first five days of reports, every one said the same thing: under a second to stage the weights, then nine to twenty-one minutes of model load. A model of 7 to 35 GB cannot be copied in one second. Staging was missing our in-region copy of the weights, and vLLM was quietly downloading the model from Hugging Face inside what we were calling model load. Nothing recorded which, because the engine’s own timing lines went to a log on the machine, and the machine was gone by the time anyone looked.
So the first change fixed no time at all. The worker started sending its runtime’s phase timings, and whether staging hit the cache, with its report. That turned up a second bug at once. Our weight cache was replicated between regions object by object, so a region could receive the file that marks a copy complete before the model files it described. A worker trusted the marker, synced half a model and failed. The marker is now a manifest that lists every file and its size, a worker checks what it synced against it, and one job fills each region and repairs any that fall short.
The weights were not the problem
With the cache working, we also moved the weights off the machine’s root disk and onto its local NVMe drive, which every AWS GPU machine we launch comes with and which reads far faster. Then the first fully instrumented boot came in, and it was still twelve minutes. The new timings said where they went.
vLLM printed nothing for five minutes after it started. In that time the root disk read 1.7 GB and spent 451 seconds doing it. The weights then loaded from the NVMe drive in 6 seconds. The slow part was Python importing vLLM and PyTorch, one small file at a time, from a disk that was not really there yet.
A machine’s root disk on AWS starts as a copy of a saved image, and the copy is lazy(opens in a new tab): each block is fetched from S3 the first time something reads it. That is fine for a server that reads its files once and then runs for months. For a machine that lives twenty minutes and reads thousands of files during its first five, it means every first read is a network round trip.
Reading ahead, and reading too much
The fix for a lazy disk is to read it before you need it, with many reads in flight at once, while the weights are staging and the disk would otherwise sit idle. The first version read everything under the Python environment and the GPU driver’s libraries: 8.8 GB in 95,611 files. That turned out to compete with vLLM for the same disk. vLLM’s first output still came three minutes in.
The second version reads only what vLLM reads. The first boot of a given vLLM build in a region records which files the engine opened, in the order it opened them, and saves the list next to the weights. Every later boot reads exactly that list: 5.1 GB in 12,791 files. vLLM’s first output now comes at 129 seconds, and the engine starts at 176, down from 301.
Compiling the same model every time
The second large block was compilation. vLLM compiles the model(opens in a new tab) for the GPU it finds, and every boot did it from scratch: about three minutes of a twelve-minute boot, producing the same result as the boot before. Now the first worker to serve a given setup (model and revision, vLLM version, GPU and serving settings) uploads its compile cache to the region, and later boots of that setup unpack it before vLLM starts.
Two rules keep it safe. A cache is written once and never replaced, and only by a worker that actually reached serving, so a broken compile cannot spread. And any failure to restore it means a normal compile, never a failed boot. Compile went from 179 seconds to 38, and the engine’s warm-up after it from 52 seconds to 4.
The result
| Phase, seconds after vLLM starts | Before | After |
|---|---|---|
| Imports, until the engine starts | 301 | 176 |
| Loading the weights | 19 | 3 |
| Compiling | 179 | 38 |
| Capturing CUDA graphs | 13 | 5 |
| Warm-up until ready | 52 | 4 |
| Ready to serving | 3 | 3 |
| vLLM start to serving | 572 | 234 |
| Launch to serving | 11.9 minutes | 7.7 minutes |
One boot before and one after is not a result, so here is every boot of that model in production. Before any of these changes, five boots took between 11.6 and 13.4 minutes from launch to serving, with a median of 12.4. Since all of them were in place, seven boots took between 5.4 and 11.9 minutes, with a median of 7.7. The three most recent took 6.1, 5.4 and 5.9.
A larger 35B model on an NVIDIA L40S moved the same way: two boots of 18.7 and 21.4 minutes before the changes, and four since, between 8.2 and 13.7 minutes with a median of 9.4.
These are still small samples, but the slow boots have a cause. The first boot of a model in a region finds no saved compile, so it compiles the model and saves the result for the next boot. Every boot since the changes that took 10 minutes or more was one of those. The boots that found a saved compile took 5.4 to 7.9 minutes on the small model, with a median of 6.1, and 8.2 and 8.3 minutes on the larger one. The sample will grow slowly from here, because most of the small model’s work now runs on machines that stay on and never boot.
What is left
- Imports are still the largest phase. The disk is read sooner, not faster, and 129 seconds pass before vLLM prints anything.
- Before vLLM starts, the machine itself takes two to four minutes to boot and stage. Some boots spend noticeably longer there than others, and I do not know why yet.
- AWS can now fill a restored disk(opens in a new tab) at a rate you pay for, from 100 to 300 MiB/s, instead of block by block on first read. We have not tried it yet; at a few gigabytes per boot it may be cheaper than the minutes it saves.
The other half of the problem is how many boots there are. A boot that never happens costs nothing, and the scheduler now waits until there is enough work to justify one, and puts small work on a larger model that is already running. That is its own post.
Guides this draws on
Before a GPU produces a token it must boot, install a serving stack and load weights. On short jobs that setup can outcost the inference, and most is removable.
Spot and remnant GPUs are the same hardware sold cheap, with the catch that it can be reclaimed. How it works, and why batch can use it when interactive cannot.
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.
The GPU a model runs on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.