Quantifying the Memory You Save Going From FP16 to Q4

Sixteen bits to four is a factor of four. The real figure for a Q4_K_M GGUF is 3.27, and the gap is not rounding — it is two specific, documented design decisions, both of which you can account for exactly.

The naive 4x

weights = P * bpw / 8

Llama 3.1 8B, 8,030,261,248 parameters:
  at 16 bits   8.03e9 * 16 / 8 = 1.606e10 bytes = 14.96 GiB
  at  4 bits   8.03e9 *  4 / 8 = 4.015e9  bytes =  3.74 GiB
  ratio 4.00x, saving 11.22 GiB

That first line is right: llama.cpp’s quantize README publishes 16.0005 bits/weight and 14.96 GiB for F16 on this model, which is exactly what the formula gives. The second line is not. The same table publishes Q4_K_M at 4.8944 bits/weight and 4.58 GiB — 0.89 bits per weight more than the name suggests, and a real ratio of 16.0005 / 4.8944 = 3.27x.

Where the first half-bit goes

Quantization is not a per-weight operation. Weights are grouped into blocks and each block carries its own scale, because a single scale across a whole tensor would be destroyed by outliers. Those scales are stored, and they count.

The k-quant formats were introduced in llama.cpp pull request #1684, merged in June 2023, whose description states the layout and the resulting bits per weight for each type. Q4_K is described there as a 4-bit type in super-blocks of 8 blocks of 32 weights, with scales and minimums quantized to 6 bits, giving 4.5 bits per weight. The same source gives Q2_K as 2.5625, Q3_K as 3.4375, Q5_K as 5.5 and Q6_K as 6.5625.

the legacy formats make the arithmetic explicit:

Q8_0   32 weights x 8 bits + one fp16 scale
       (32*8 + 16) / 32 = 8.5 bits per weight
       llama.cpp publishes 8.5008 for this type. Exact.

Q4_0   32 weights x 4 bits + one fp16 scale
       (32*4 + 16) / 32 = 4.5 bits per weight

So half a bit per weight is metadata, in every four-bit format, before any tensor is treated specially. 16 / 4.5 = 3.56x, not 4x.

The overhead is a fixed cost per block, so it is entirely a function of block size. A 32-weight block with one 16-bit scale spends 16 bits on metadata whether the weights are 8 bits each or 4; at 8 bits that is a 6% overhead and at 4 bits it is 12.5%. This is the reason the proportional penalty gets worse as you quantize harder, and the reason the k-quants use super-blocks: sharing a second-level scale across eight blocks amortises part of the metadata over 256 weights instead of 32. Going below four bits without that structure would spend more on scales than on weights.

Smaller blocks are not simply wasteful, though, and the trade is the whole design problem. A scale covers a group of weights, and one outlier inside the group forces the scale wide enough to represent it, which costs precision for every other weight sharing that scale. So a smaller block is more accurate and more expensive, and each format is a choice of where to sit on that curve. It is also why quantization damage is concentrated where outlier magnitudes are: a tensor with a few extreme activations quantizes worse than one with a smooth distribution, which is exactly the observation the mix below is built on.

Where the second half-bit goes

The remaining 0.39 bits is the mix. The _M and _S suffixes are not different block layouts; they are policies about which tensors get a better one. The documented policy for Q4_K_M is to use Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest, with the output tensor also kept at Q6_K.

The choice of tensors is not arbitrary. The value projection and the feed-forward down-projection are the two places where activations entering the multiply have the widest dynamic range, so they are the two that lose most under a coarse scale; the output head is upgraded because its errors land directly on the logits with nothing after them to absorb the damage. Upgrading those and leaving everything else at four bits is a deliberate purchase of quality with a known number of bits, which is why the _S and _M variants of the same nominal type differ in size at all.

That is enough to reconstruct the number. For Llama 3.1 8B — 32 layers, hidden 4096, 8 KV heads at head dimension 128, feed-forward intermediate 14336, vocabulary 128256:

per layer, the two upgraded tensors:
  wv (attention value)   4096 * 1024   =    4,194,304
  w2 (ffn down)          4096 * 14336  =   58,720,256
                                          -----------
                                           62,914,560
  half of that, x 32 layers             = 1,006,632,960
  plus the output tensor 128256 * 4096  =   525,336,576
                                          --------------
  at Q6_K (6.5625 bpw)                    1,531,969,536
  everything else, at Q4_K (4.5 bpw)      6,498,291,712
                                          --------------
  total                                   8,030,261,248

Checking the reconstruction

bits = 1,531,969,536 * 6.5625  +  6,498,291,712 * 4.5
     = 1.0054e10           +  2.9242e10
     = 3.9296e10 bits

bits per weight = 3.9296e10 / 8,030,261,248 = 4.8935

llama.cpp publishes                            4.8944
error                                          0.02%

Two independently derived quantities — a parameter count computed from the model’s shapes, and a bits-per-weight computed from a block layout and a tensor policy — reproduce a published measurement to two decimal places. Running the same reconstruction for Q5_K_M, which applies the identical mix over a 5.5-bit base, gives 5.7027 against a published 5.7036.

The residual is worth naming rather than waving at. Roughly a thousandth of a bit per weight remains unaccounted for, and it is the GGUF container: the metadata block, the tokenizer, the tensor directory and the alignment padding, all of which are in the file size the README reports and none of which is a weight. On an 8 billion parameter model a thousandth of a bit is about a megabyte, which is the right order for a vocabulary of 128,256 entries stored as strings. That the discrepancy points the right way and is the right size is a better check than the agreement itself.

Which means the shortfall is fully accounted for and none of it is waste:

4.00 bits   the name
+0.50       block scales and minimums (k-quant super-block layout)
+0.39       the _M mix: value and down-projection tensors, and the output head
-----
 4.89 bits  published

16.0005 / 4.8944 = 3.27x, not 4x
Llama 3.1 8B: 14.96 GiB -> 4.58 GiB, saving 10.38 GiB

And the cache does not shrink at all

The last correction is the one that matters most in practice. Quantizing weights does nothing to the key/value cache, which is stored separately and at its own precision. So the ratio you actually experience falls as the context grows:

Llama 3.1 8B, fp16 KV cache at 128 KiB/token:

context      F16 total      Q4_K_M total     ratio
      0      14.96 GiB       4.58 GiB        3.27x
  8,192      15.96           5.58            2.86x
 32,768      18.96           8.58            2.21x
131,072      30.96          20.58            1.50x

The reason the ratio degrades is that quantizing the weights shrinks the numerator and the denominator of the total by the same absolute amount while leaving the cache untouched in both, so as the cache grows it dilutes the saving from both sides. In the limit where the cache dominates, the ratio approaches one and quantization saves you nothing at all. Nothing about that is a defect of the format — the cache is a different kind of object and is quantized by a different flag — but it means the headline compression figure describes a case that only exists at zero context.

At the model’s full context window, four-bit weights save you a third of the memory rather than three quarters of it. If you are quantizing in order to fit a long context, this is the table to size against, and the cache is the term to attack next — moving it to q8_0 halves it, as worked through on the 8 GiB budget page. What quantization costs in quality, rather than in bytes, is a separate question covered under quantization at inference time.