Infrastructure

GPUs for AI inference: T4, L4, L40S, A100, H100

Pick the GPU wrong and you either cannot fit the model or you pay for a supercomputer to run a task a modest card would handle. Memory is the first constraint; price is the second.

9 min read Updated August 19, 2026

Memory is the first gate

Before performance or price, a GPU has to physically hold the model. The weights (parameter count times bytes-per-weight after any quantization), plus working memory for activations and the KV cache, must fit in the GPU memory (VRAM). If they do not, the model cannot run on that card at all — or must be split across several, which adds cost and complexity.

So the first question for any model is: how much VRAM does it need? An 8B model at 4-bit needs only a handful of gigabytes; a 70B model at full precision needs well over a hundred and must span multiple large GPUs. Quantization (see the quantization guide) is the main way to move a model down to a cheaper card.

The common inference cards

GPU	VRAM	Tier	Best for
T4	16 GB	Cheapest	Small models (up to ~7-8B quantized); the value floor for light batch work.
L4	24 GB	Low	Small-to-mid models; an efficient, inexpensive modern inference card.
L40S	48 GB	Mid	Mid-size models (up to ~24-32B quantized) with headroom; strong all-rounder.
A100	40 / 80 GB	High	Large models and high throughput; the previous-generation workhorse.
H100	80 GB	Highest	The largest models and maximum throughput; newest mainstream data-center GPU.

Memory and price climb together. The cheapest card that fits your model with headroom is usually the right one.

Newer generations also serve newer numeric formats (for example, the latest 4-bit float formats need the newest hardware), so the card sets not just how big a model you can run but which quantized builds you can run efficiently.

Matching a model to a card

The goal is the cheapest card that holds your model with comfortable headroom and serves your throughput. A worked path:

Compute the VRAM footprint: parameters x bytes-per-weight (after quantization) x ~1.2 for overhead.
Find the cheapest card whose VRAM exceeds that with room for the KV cache at your batch size.
Confirm the card supports your model quantization format.
If nothing affordable fits, quantize further or pick a smaller model before reaching for a bigger GPU.

There is a second, less obvious consequence of memory being the gate, and it decides which card is actually cheaper per token. Decoding is bound by how fast weights can be read out of memory, not by arithmetic — the point Shazeer made when introducing multi-query attention, where incremental decoding is described as slow because of the memory-bandwidth cost of repeatedly loading the large key and value tensors. So the figure that predicts cost per million tokens is memory bandwidth per dollar, and it does not track VRAM, price, or generation reliably. A newer, more expensive card can deliver fewer tokens per dollar than an older one it comfortably beats on every spec-sheet headline. Check the bandwidth number, not the tier name.

The spot-market angle

On the spot market, the cheapest card for a model is not fixed — it moves with supply and demand across regions and over time. A platform that shops capacity will sometimes find a normally-pricier card available cheaply enough to beat the usual choice. Defining a set of acceptable instance types per model (rather than a single one) lets the scheduler take whichever is cheapest at run time. This is why the model catalog maps each model to candidate instance types, not just one.

Sources

Sources and further reading

Fast Transformer Decoding: One Write-Head is All You Need(opens in a new tab)
Shazeer, 2019. Why decode is memory-bandwidth-bound — the reason bandwidth per dollar, not FLOPS, predicts your cost per token.
Efficiently Scaling Transformer Inference(opens in a new tab)
Pope et al., 2022. An analytical model for choosing a partitioning strategy given a latency target, and the latency / MFU frontier that results.
Efficient Memory Management for LLM Serving with PagedAttention(opens in a new tab)
Kwon et al., SOSP 2023. The reason leftover VRAM converts into throughput rather than sitting idle.
The concurrency cliff
What the headroom on a given card is actually worth, and what happens when you spend more of it than you have.

Frequently asked questions

How much GPU memory do I need to run a model?

Roughly parameter count times bytes-per-weight (2 for BF16, 1 for FP8/INT8, 0.5 for INT4) times about 1.2 for overhead, plus room for the KV cache. An 8B model at 4-bit needs only a few GB; a 70B model at full precision needs over 100 GB across multiple GPUs.

Which GPU is cheapest for inference?

For small models, a T4 (16 GB) is the value floor, with the L4 (24 GB) a more modern low-cost option. The right answer is the cheapest card that fits your model with headroom for the KV cache — and on the spot market that can shift between cards over time.

Raising batch concurrency lifts throughput until suddenly it does not. GPU memory, not compute, sets the ceiling, and crossing it makes requests fail, not slow.

Quantization explained: FP8, INT4, and serving models cheaper

Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.

Mixture-of-Experts, explained: big-model quality at small-model cost

An MoE model holds many parameters in memory but activates a fraction per token. Total drives memory, active drives speed, and that split changes the economics.

What an LLM inference request really costs

Behind every per-token price is a GPU running for some number of seconds. The GPU-seconds model of inference cost, and the two levers that actually move it.