Quantization explained: FP8, INT4, and serving models cheaper

What quantization does

A model is a very large set of numbers, called weights. Normally each one is stored in 16 bits (two bytes); the formats are named BF16 and FP16. Quantization stores them with fewer bits: 8 bits (one byte), or even 4 (half a byte). Fewer bits per weight means the model needs less GPU memory and the GPU can read it faster, so you need cheaper hardware and pay less per token.

The biggest effect is on memory. A model needs roughly its number of weights times the bytes per weight, plus some working space for the requests it is serving. Halve the bytes per weight and you roughly halve the memory:

FP8 vs. INT8: the same size, spent differently

FP8 and INT8 both store a weight in one byte, so neither is smaller: a model with 8 billion weights needs about 8 GB in either, plus working space for the requests it serves. The difference is what the byte holds.

The spacing is what decides quality. A model carries a few values far larger than the rest (the outliers LLM.int8() was built around). FP8 represents them alongside the small values with little extra rounding; INT8 needs extra handling, such as keeping the outliers in 16 bits, to do as well. Done well, both land close to the 16-bit model.

Which to choose depends mostly on the GPU. On one with FP8 hardware, FP8 is usually the simpler choice, with no outlier handling to tune. On an older GPU, INT8 runs at full speed and FP8 does not: serving software such as vLLM still loads FP8 weights there, saving the memory, but does the arithmetic in 16 bits.

On a 16 GB card, an 8-bit model of about 8 billion weights fits with room for long prompts. One of 14 billion does not fit at 8 bits, but does at 4.

What it costs in quality

Quantization loses a little information: storing each weight in fewer bits means rounding it, and rounding adds small errors. Modern methods are very good at keeping those errors from mattering. The skill is in how the rounding is done.

It is worth knowing why, because "compression that loses almost nothing" sounds like marketing until you see the mechanism. A small minority of weights do a large share of the work, and every good method is built around finding and protecting them. The AWQ paper puts it plainly:

Protecting only 1% salient weights can greatly reduce quantization error.

Their second finding is the surprising half: to find those weights, "we should refer to the activation distribution, not weights". In plain terms, which weights matter depends on the text running through the model. So the sample text a quantization is tuned on (its calibration data) is part of the method: one tuned on text unlike yours may do worse on your work than one tuned on text like it.

8-bit (FP8, INT8)

The quality loss is usually too small to measure against the 16-bit model. A safe default for cheaper serving.

4-bit (INT4 via AWQ, GPTQ, NVFP4, MXFP4)

Bigger savings, with a small but real quality cost. Good methods tune on sample text to protect the weights that matter most, which keeps the loss minor for most tasks.

Whether the 4-bit cost matters depends on your task, and only a test on your own data can say. For many classification and extraction jobs the difference does not show; for the hardest reasoning it can. The usual pattern: compress hard, then check on a gold set.

The method names you will see

Quantized models on model-sharing sites carry tags that name the format. A quick key:

• AWQ, GPTQ: popular 4-bit methods that tune on sample text to protect quality. Widely supported.
• FP8, W8A8: 8-bit formats. FP8 keeps quality high and runs well on recent GPUs.
• NVFP4, MXFP4: newer 4-bit formats. NVFP4 in particular needs the latest GPU generation.
• BNB, 4bit: the bitsandbytes library, often used because it is convenient.

Why it matters for batch cost

Quantization moves a model onto cheaper hardware. A model that needs an expensive 80 GB GPU at 16 bits might fit comfortably on a much cheaper 24 GB card at 4 bits. A cheaper card that reads the model faster means a lower cost per token. Together with batching and spare (spot) capacity, it is one of the main ways a provider lowers its costs without changing what you see.

On Otium you do not choose a format. You choose a size (small, medium or large), and we pick and test the model and precision behind it. What you should still test is the size, on your own gold set.

Sources

The method names in this guide are all papers. If you are choosing between quantized builds of the same model, the differences between them are in these four papers; the vLLM page covers which GPUs run FP8.