GPUs for AI inference: T4, L4, L40S, A100, H100
Which GPU should a model run on? The cheapest one that holds it with room to spare. Pick wrong and either the model does not fit, or you pay for a top-end card to do work a modest one could handle. Memory decides first; price second.
Memory comes first
Before speed or price, a GPU has to hold the model. Its memory (called VRAM) must fit the weights, which is roughly the number of parameters times the bytes per weight, plus working space for the requests in flight. The biggest part of that working space is the KV cache, where the model keeps what it has already read of each request. If it all does not fit, the model cannot run on that card, or has to be split across several, which costs more and adds complexity.
So the first question for any model is how much memory it needs. An 8-billion-parameter model stored in 4 bits needs only a few gigabytes; a 70-billion-parameter model at full 16-bit precision needs well over a hundred, spread over several large GPUs. Compressing the model (quantization) is the main way to move it onto a cheaper card.
The common inference cards
| GPU | VRAM | Price | Best for |
|---|---|---|---|
| T4 | 16 GB | Cheapest | Small models (up to about 7 to 8 billion parameters, compressed); the floor for light batch work. |
| L4 | 24 GB | Low | Small and mid-sized models; an efficient, inexpensive modern card. |
| L40S | 48 GB | Mid | Mid-sized models (up to about 24 to 32 billion parameters, compressed) with room to spare; a strong all-rounder. |
| A100 | 40 / 80 GB | High | Large models and high volume; the previous generation’s workhorse. |
| H100 | 80 GB | Highest | The largest models and the most throughput; the newest mainstream data-center GPU. |
Newer generations also run newer compact number formats (the latest 4-bit formats need the newest hardware). So the card decides not only how big a model you can run, but which compressed versions of it run fast.
What each of these cards costs per hour right now, by cloud and region, is on our live GPU price table.
Matching a model to a card
You want the cheapest card that holds your model comfortably and handles your volume. Step by step:
- Work out the memory the model needs: parameters × bytes per weight (after any compression) × about 1.2 for overhead.
- Find the cheapest card with more memory than that, leaving room for the KV cache at the number of requests you run at once.
- Check the card runs your model’s compressed format well.
- If nothing affordable fits, compress further or choose a smaller model before reaching for a bigger GPU.
There is a second, less obvious effect, and it decides which card is actually cheaper per token. Writing an answer is limited by how fast the GPU can read the weights out of memory, not by how fast it can calculate. Shazeer made the point when introducing multi-query attention: generating text one token at a time is slow because of the memory-bandwidth cost of repeatedly loading the large key and value tensors. So the number that predicts cost per million tokens is memory bandwidth per dollar. It does not reliably follow memory size, price or generation: a newer, pricier card can produce fewer tokens per dollar than an older one it beats on every headline spec. Check the bandwidth figure, not the name.
On the spot market, the cheapest card moves
On the spot market, where clouds sell spare capacity at a discount, the cheapest card for a model is not fixed. It moves with supply and demand, from region to region and hour to hour. Sometimes a normally pricier card is cheap enough to beat the usual choice.
That is why Otium lists several acceptable machine types for each model size rather than one, and takes whichever is cheapest when your work runs. You choose a size and a deadline; the card is our problem. Our monthly GPU market report shows how much the same card’s price varies by region.
Sources
Frequently asked questions
How much GPU memory do I need to run a model?
Roughly the number of parameters times the bytes per weight (2 at 16-bit, 1 at 8-bit, 0.5 at 4-bit), times about 1.2 for overhead, plus room for the KV cache. An 8-billion-parameter model at 4 bits needs a few gigabytes; a 70-billion-parameter model at 16 bits needs over 100 GB across several GPUs.
Which GPU is cheapest for inference?
For small models, the T4 (16 GB) is the cheapest floor, with the L4 (24 GB) a more modern low-cost option. The right answer is the cheapest card that fits your model with room for the KV cache, and on the spot market that changes from card to card over time.
Related guides
Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.
An MoE model holds many parameters in memory but activates a fraction per token. Total drives memory, active drives speed, and that split changes the economics.
LLM inference cost per token: list prices per million tokens, then how GPU-seconds become that price, what moves throughput, and a worked example.