Quantization explained: FP8, INT4, and serving models cheaper

What quantization does

A model is a large pile of numbers (weights). By default those are stored at 16-bit precision (BF16/FP16) — two bytes each. Quantization stores them at lower precision: 8 bits (one byte) or even 4 bits (half a byte). Fewer bits per weight means the model takes less GPU memory and moves through the memory system faster, which lowers both the hardware you need and the cost per token.

The headline effect is on memory. A model footprint is roughly parameter count times bytes-per-weight (plus overhead for activations and the KV cache). Halving the bytes-per-weight roughly halves the memory:

What it costs in quality

Quantization is lossy: representing weights with fewer bits introduces small errors. The good news is that modern quantization methods are remarkably good at hiding this. The art is in how the rounding is done.

It is worth understanding why, because "lossy compression that does not lose much" sounds like marketing until you see the mechanism. The errors are not spread evenly: a small minority of weights carry a disproportionate share of the model's behaviour, and every good method is built around finding and protecting them. AWQ states the finding baldly:

Protecting only 1% salient weights can greatly reduce quantization error.

Their second finding is the counter-intuitive half: to find those weights, "we should refer to the activation distribution, not weights". Which weights matter depends on what flows through them, so calibration data is part of the method rather than a nicety — and a quantization calibrated on text unlike yours is a different proposition from one calibrated on text like yours.

8-bit (FP8, INT8)

Quality loss is usually negligible — often within measurement noise of the full-precision model. A safe default for cheaper serving.

4-bit (INT4 via AWQ, GPTQ, NVFP4, MXFP4)

Bigger savings, small but real quality cost. Smart methods calibrate on sample data to preserve the weights that matter most, keeping the loss minor for most tasks.

Whether the 4-bit quality cost matters is, again, a measurement against your task — not a universal verdict. For many classification and extraction jobs the difference is invisible; for the hardest reasoning it can show. The pattern is to quantize aggressively, then verify on a gold set.

The method names you will see

Quantized models on model hubs carry tags that tell you the format. A quick decoder:

• AWQ, GPTQ — popular 4-bit weight-quantization methods that calibrate on data to preserve quality. Widely supported.
• FP8, W8A8 — 8-bit formats; FP8 is a floating-point 8-bit type with strong quality and good hardware support on recent GPUs.
• NVFP4, MXFP4 — newer 4-bit floating-point formats; NVFP4 in particular targets the latest GPU generation.
• BNB / 4bit — bitsandbytes quantization, often used for convenience.

Why it matters for batch cost

Quantization moves a model down the hardware ladder. A model that needs an expensive 80GB GPU at full precision might fit comfortably on a much cheaper 24GB card once quantized to 4-bit — and cheaper hardware plus faster memory throughput compounds directly into a lower cost per token. Combined with batching and spot capacity, it is one of the main levers a platform uses to drive the cost basis down without changing what the customer sees.

Sources

The method names in this guide are all papers. If you are choosing between quantized builds of the same model, the differences between them are in these four.