Why 4-Bit Quantization Became the Local-Inference Default

Four bits is not a compromise that happened to work. It is the point where three unrelated constraints — how much memory a bit is worth, where the published quality curve turns down, and what a byte divides into — all point at the same number. That coincidence is why it settled so quickly and why it has stayed settled.

The claim

The argument here is that 4-bit weight-only quantization is the local default for reasons that are mostly structural, and that anyone treating it as a quality ranking has the causation backwards. Nobody established that 4 bits is the highest fidelity per byte; what happened is that 4 bits is where the memory saving stops being dramatic, where the accuracy cost starts being dramatic, and where the hardware is convenient — and a format needs all three.

This matters practically because it tells you when to move off it. If 4 bits won on a quality argument, deviating would be a quality decision. Because it won on three arguments, moving off it means checking which of the three you are actually free of.

The size curve has a knee, and it is at 4

Memory is a hyperbola in bit width, not a line, and the consequence is that each bit you remove is worth less than the last. Take a 13B model and count only the linear weights, ignoring the per-group scale overhead so the shape is visible:

bits   size          saved vs. FP16   saved vs. previous row
16     26.0 GB       —                —
8      13.0 GB       13.0 GB          13.0 GB
6       9.75 GB      16.3 GB           3.25 GB
5       8.13 GB      17.9 GB           1.63 GB
4       6.50 GB      19.5 GB           1.63 GB
3       4.88 GB      21.1 GB           1.63 GB
2       3.25 GB      22.8 GB           1.63 GB

The absolute step is constant, so the interesting column is the fraction of what is left. Going 16 → 8 removes half the model. Going 8 → 4 removes half of what remains, which is another quarter of the original. Going 4 → 2 removes half again, which is only an eighth of the original — and by then the KV cache and the runtime are a large share of your memory, so an eighth of the weights is a much smaller fraction of your actual footprint.

Concretely: dropping from 4 bits to 3 on that 13B saves 1.63 GB. On a 24 GB card already holding 6.5 GB of weights and several gigabytes of KV cache, 1.63 GB is a modest extension of context length. Dropping from 8 to 4 saved 6.5 GB, which is the difference between the model fitting and not. The purchase gets less valuable exactly as it gets more expensive, and the crossing point is around 4.

What the published numbers say about 3 bits

The other half of the argument needs evidence rather than arithmetic, and there is a clean published source for it. The GPTQ paper (ICLR 2023) reports WikiText2 perplexity for OPT models at FP16, 4 bits and 3 bits under its ungrouped per-row setting. Three rows, quoted exactly:

OPT model   FP16     GPTQ 4-bit   GPTQ 3-bit
13B         10.13    10.31        11.61
30B          9.56     9.63        10.27
175B         8.34     8.37         8.68

At 4 bits the deltas are 0.18, 0.07 and 0.03. At 3 bits they are 1.48, 0.71 and 0.34 — between five and twelve times larger for one further bit. That is not a smooth curve; it is a bend, and it is in the same place in every row.

Two honest caveats before leaning on it. These are 2023 numbers on the OPT family, which is old, and the setting is ungrouped — a modern 4-bit bake with group size 128 has smaller deltas than the table shows, and a modern 3-bit bake with a small group size does better than 3-bit here too. And perplexity on WikiText2 is not your task. The argument the table supports is about the shape: whatever your model and your evaluation, the marginal bit below 4 costs disproportionately more than the marginal bit above it. That claim is examined further in INT4 accuracy loss by model size.

The reason that has nothing to do with quality

Four divides eight. Two 4-bit weights pack into one byte exactly, with no value straddling a boundary and no shift-and-mask across two loads. Unpacking is a shift and an AND on a value the kernel already has in a register.

Three bits does none of that. Eight 3-bit weights occupy three bytes, so weight boundaries fall at bit offsets 0, 3, 6, 9, 12 — half of them crossing a byte edge. A kernel reading them must either handle the straddling case with extra instructions on every load, or adopt a padded layout that wastes the bit it was trying to save. Five bits has the same problem in the other direction, and both are why 3-bit and 5-bit formats have historically had thinner kernel support and narrower hardware coverage than 4-bit — a practical constraint the quality tables never show.

It compounds. Because 4-bit had the best kernel support, it got the most optimisation effort, which produced kernels like Marlin that made 4-bit fast at batch sizes where other widths were not, which made 4-bit the width people published checkpoints in, which made it the width the next kernel targeted. Formats that avoid the alignment problem by construction — GGUF’s super-blocks, EXL2’s mixed widths — exist partly because they had to solve packing themselves to escape it.

The strongest case against

The position above is defensible but not universal, and the honest version has to state where it breaks.

• The knee moves with model size. The GPTQ table shows the 3-bit penalty shrinking as models grow — 1.48 at 13B, 0.34 at 175B. Extrapolating a trend from a 2023 table is exactly the error this cluster warns about, but the direction is consistent with the redundancy argument, and it means the case for 4 bits is strongest on small models and weakest on very large ones. If you are running something enormous locally, 3 bits deserves a real evaluation rather than a dismissal.
• Uniform bit width is the wrong frame anyway. The comparison “4 bits versus 3” assumes every weight gets the same budget, which none of the good formats actually do. GGUF’s M mixes, EXL2’s solved allocation and mixed-precision quantization all spend a fractional average, and a well-allocated 3.5 bpw can beat a uniform 4.
• Non-uniform codebooks change the arithmetic. NF4 gets more out of sixteen levels than int4 does by placing them on quantile boundaries. The bit budget is a container, and how much fits in it is not fixed.
• Hardware moves. The byte-alignment argument is a property of current instruction sets. Native support for other widths, or for block floating-point formats, weakens the third leg of the argument without touching the first two.