Weight-Only Quantization Explained

Nearly every quantized model you can download for local use is weight-only: the weights are 4 bits, the activations are still 16. That is not a compromise anyone settled for. It is the correct answer to the specific problem local inference has, and the arithmetic says why.

What W4A16 actually names

The notation is two numbers: bits for weights, bits for activations. W4A16 means 4-bit weights and 16-bit activations. W8A8 means both at 8 bits. W16A16 is the unquantized baseline.

The second number is the one that decides what kind of technique you are looking at. If the activations are quantized, the matrix multiply itself changes — it becomes an integer operation on integer hardware, and everything about outliers, calibrated ranges and clipping applies. If they are not, the multiply is unchanged and quantization is purely a storage decision that gets undone in the kernel before any arithmetic happens. GPTQ, AWQ, NF4, GGUF’s K-quants and EXL2 are all weight-only. SmoothQuant and LLM.int8() are not.

The memory arithmetic

Weights are a fixed cost paid once and held for the life of the process. Activations are transient, sized by batch and sequence length. For a single-user local setup those two quantities are not remotely comparable, and that asymmetry is the argument.

Take a 13B model at 4.156 bits per weight (4-bit, group size 128 — see group size for where the 0.156 comes from) versus FP16:

weights, FP16    13e9 * 16 / 8   = 26.0e9 bytes = 26.0 GB
weights, W4      13e9 * 4.156 / 8 = 6.75e9 bytes = 6.75 GB
saved                                            = 19.2 GB

one layer's activations, batch 1, one token, FP16:
  hidden 5120 * 2 bytes                          = 10.2 kB
same, quantized to int8:
  hidden 5120 * 1 byte                           = 5.1 kB
saved                                            = 5.1 kB

Nineteen gigabytes against five kilobytes. Quantizing the activations of a single-stream decode saves a rounding error’s worth of memory while introducing every failure mode that outlier features bring with them. There is no version of that trade that is worth taking.

The picture only changes when activations get large, and they get large in exactly two situations: a long prefill, where the whole prompt is processed at once, and a large serving batch. Both are the throughput regime, not the local regime.

The memory that does compete with the weights locally is the KV cache, and it is worth sizing because it changes what quantizing the weights was for. For the same 13B — 40 layers, 40 heads of dimension 128, no grouped-query reduction — the cache holds a key and a value per layer per token:

bytes per token = 2 (K and V)
                * 40 layers
                * 40 heads * 128 head_dim
                * 2 bytes (fp16)
                = 1.64e6 bytes ≈ 1.64 MB per token

  4,096 tokens →  6.7 GB
 16,384 tokens → 26.8 GB
 32,768 tokens → 53.7 GB

At 16k of context the cache is four times the size of the quantized weights. So on a single card, quantizing the weights is not only about fitting the model — it is about how much context is left over once the model is in, and past a certain sequence length the cache is the binding constraint no matter what the weights cost. Models with grouped-query attention cut the per-token figure by the ratio of query heads to key/value heads, which is why that architectural choice matters far more to local users than its description suggests.

Why the activations are left alone

Beyond the memory being negligible, the activations are the hard operand. Weights are static, known offline, and can be quantized at any granularity you like because their scales are applied during unpacking. Activations are produced at runtime, contain systematic outlier features from around 6.7B parameters upward, and can only carry scales along axes that survive the accumulation.

Leaving them at 16 bits does not merely avoid work. It sidesteps the entire outlier problem: nothing is clipped, no calibrated range can be exceeded by an unusual input, and a prompt in a language the calibration corpus never contained cannot push an activation outside a range somebody measured last month. Weight-only quantization is therefore the variant with no input-dependent failure mode, which is a large part of why it is what gets distributed.

What it costs

• Extra arithmetic, not less. The kernel unpacks and rescales every weight before the multiply. It is doing everything the FP16 kernel did plus dequantization, and it wins only because the multiply was waiting on memory anyway. The mechanism is worked through in the Marlin kernel.
• No integer-tensor-core speedup. W8A8 gets to run the multiply on integer units at roughly twice FP16 throughput. W4A16 does not — the multiply is still FP16 and always will be.
• Quality loss that is real but bounded. Rounding the weights changes the model’s function. How much depends on the method, the bit width and the model size; the published evidence and the reason the loss shrinks with scale are in INT4 accuracy loss by model size.
• Kernel dependence. The advantage is entirely a property of the kernel you get, not the file you downloaded. The same checkpoint can be fast or unremarkable depending on which path the runtime selects on your card.

Where it stops being the right answer

The crossover is arithmetic intensity. Each loaded weight is used once per sequence in the batch, so batch size B gives roughly B multiply-accumulates per weight loaded. Against a FLOP-to-byte ratio of 100 to 200 on current accelerators, a bandwidth-bound kernel stops being bandwidth-bound somewhere in the tens — and published kernel results put the practical edge of the useful window at batch sizes in the region of 16 to 32.

Above that, four things are true at once: the multiply is compute-bound, the dequantization is added work on the critical path, the activations are now large enough to be worth quantizing, and the KV cache rather than the weights is what is filling the card. That is the regime W8A8 and cache quantization were designed for. Below it — one person, one stream, one card — weight-only is not a compromise, it is the right shape of solution.