Quantisation for Inference: FP16, INT8, FP8 and INT4

Quantisation is the only lever that improves both the cost and the speed of decoding at the same time, which is why every serving stack reaches for it. What it costs in quality is the part where the honest answer is “it depends, measurably, and here is how the published work frames it”.

Why anybody quantises at all

Decode is bandwidth-bound: each step reads the entire weight set from memory to produce one token. Cut the bytes per weight and you cut the binding constraint proportionally. Half the bytes is, to a first approximation, half the decode time and half the memory footprint — and the memory saving is often the more consequential of the two, because it decides whether the model fits on one card at all, and how much space is left for the KV cache and therefore how many users fit.

The formats

Two distinctions matter and are routinely blurred. Weight-only quantisation stores weights small and computes in higher precision after dequantising; it fixes bandwidth, which is what decode needs. Weight-and-activation quantisation also does the arithmetic in the low format, which additionally raises throughput on hardware with native support. And post-training quantisation (GPTQ, AWQ) is applied to a finished checkpoint using a small calibration set, whereas quantisation-aware training bakes it in during training at much greater cost.

The arithmetic, exactly

Unlike quality, the resource side is not in dispute. For a dense model with P parameters:

weight bytes = P * bytes_per_weight

70B model:
  BF16   70e9 * 2    = 140 GB   -> 2x 80GB cards minimum, no room for much cache
  FP8    70e9 * 1    =  70 GB   -> fits one 80GB card, ~6 GB left for KV
  INT4   70e9 * 0.5  =  35 GB   -> one card, ~40 GB of KV budget

decode speed ceiling ~= memory_bandwidth / weight_bytes
  3.35 TB/s / 140 GB  ~=  24 steps/s      (H100 SXM bandwidth, batch of 1)
  3.35 TB/s /  35 GB  ~=  96 steps/s

The bandwidth figure is NVIDIA’s published H100 SXM specification; the ceilings are upper bounds that ignore attention, KV reads and kernel inefficiency, so real rates are lower. But the ratio is the point: 4× fewer weight bytes is a 4× higher ceiling on single-stream decode, and the entire difference between “needs two cards” and “needs one, with room for users”.

Note the second-order effect in the last column above. INT4 does not just make the model smaller; it hands the freed memory to the KV cache, which multiplies concurrency, which is where the per-token cost saving actually comes from.

One more distinction that the headline format hides: quantisation is rarely applied uniformly. Production recipes routinely keep the embedding table and the output projection at higher precision, because errors there affect every token rather than one layer’s contribution; they keep normalisation parameters in 16-bit because they are tiny and sensitive; and they use per-channel or per-group scales rather than one scale per tensor, because a single outlier weight otherwise sets the scale for thousands of ordinary ones. So “INT4 model” describes most of the weights, not all of them, and the real memory footprint is somewhat above the naive P × 0.5 — usually by a few per cent once the scales themselves are counted.

What the literature claims

Here is where a tidy table would be dishonest. Published quantisation results are reported per model, per method, per calibration set and per benchmark, and they do not compose: an INT4 number for one 70B model under AWQ says little about a different architecture under GPTQ. What can be reported faithfully is the shape of the claims:

• 8-bit is the settled case. LLM.int8() (Dettmers et al., NeurIPS 2022) reports matching 16-bit performance on models up to 175B once outlier feature dimensions are kept in higher precision, and identifies those outliers as the reason naive INT8 had failed at scale. SmoothQuant (Xiao et al., ICML 2023) attacks the same problem by migrating activation difficulty into the weights.
• 4-bit is good but not free. GPTQ reports 3- and 4-bit post-training quantisation of 175B-class models with what it characterises as negligible perplexity degradation; AWQ reports improvements over GPTQ by protecting the small fraction of salient weights identified from activation statistics. Both papers report degradation growing as models get smaller — a 7B model has less redundancy to give up than a 175B one.
• Perplexity is the wrong yardstick for your decision. Most published tables lead with perplexity because it is cheap and comparable. Degradation shows up unevenly across tasks, and the literature repeatedly finds that long-form generation, code, and multi-step reasoning are more sensitive than the aggregate number suggests.
• The KV cache can be quantised separately. An FP8 KV cache halves the per-token memory in the cache formula and is an independent decision from the weight format.

Choosing, and the part you must do yourself

A defensible default: BF16 when quality is the constraint and you can afford the memory; FP8 as the general-purpose serving format on hardware that supports it natively; INT4 when memory or cost is the binding constraint and the model is large enough to absorb it. For models under about 10B parameters, treat aggressive quantisation as something to verify rather than assume.

The part no page can do for you: run your own evaluation set at both precisions. Not perplexity — your actual task, scored the way you score it in production, on enough examples that the difference you care about is larger than the noise. Quantisation damage is task-shaped, and the only benchmark that predicts your outcome is yours.

One further wrinkle: when you call a hosted API you frequently are not told which format is being served, and it can change. That problem has its own page — detecting whether the endpoint changed underneath you.