Why Two Quantization Tools Produce Different File Sizes for the Same Model

Two files, same model, same architecture, both labelled Q4_K_M, and one is several hundred megabytes larger. Neither is corrupt and neither is mislabelled. The label names a quantization recipe, and the recipe has parameters that whoever produced the file was free to set differently.

Q4_K_M names a mixture, not a format

A k-quant type is not applied uniformly. The “M” in Q4_K_M stands for medium, and what it selects is a mixture: most tensors at Q4_K, some at Q6_K, some left larger, chosen by which tensors the quantization scheme judges most sensitive. Q4_K_S is the same base type with a less generous mixture.

Two things follow immediately. First, the average bits per weight of a Q4_K_M file is not 4 — the llama.cpp quantize README’s own table gives 4.8944 bits per weight for Q4_K_M on Llama-3.1-8B, against 8.5008 for Q8_0. The mixture is most of that gap between the nominal and the actual. Second, the mixture is code, and code changes: llama.cpp has revised which tensors get promoted more than once, so the same command run against the same input on two versions of the tool produces two different files.

llama-quantize --pure exists precisely to turn this off: its documented behaviour is to “disable k-quant mixtures and quantize all tensors to the same type”. A --pure Q4_K file is markedly smaller than a Q4_K_M one and is a different artifact with the same-looking name.

The two tensors that move the number most

The token embedding matrix and the output projection are the two largest single tensors in most decoder models, and they are the two most often given a special quantization type. It is worth deriving how much they are worth.

Llama 3.1 8B has a vocabulary of 128,256 and a hidden size of 4,096, so token_embd.weight holds 128,256 x 4,096 = 525,336,576 parameters. The 8B does not tie its output projection to its embeddings, so output.weight is a second tensor of the same shape. Together that is roughly 1.05 billion parameters — about 13% of the model’s 8.03 billion — sitting outside the transformer layers entirely.

Cost of moving ONE of those two tensors from Q4_K to Q8_0:

  525,336,576 params x (8.5 - 4.5 bits) / 8 bits per byte
    = 525,336,576 x 0.5 bytes
    = 262,668,288 bytes
    ~ 250 MiB

Both tensors: ~500 MiB, from a decision that changes nothing
inside any transformer layer.

(Assumes 8.5 bpw for Q8_0, measured, per the llama.cpp README;
 and 4.5 bpw as the nominal figure for Q4_K.)

That derivation is most of the answer to the question in the title. Two producers who disagree about whether the embeddings deserve Q8_0 produce files a quarter of a gigabyte apart on an 8B, and further apart on a model with a larger vocabulary. The flags are documented: --token-embedding-type sets “a specific quant type for the token embeddings tensor”, --output-tensor-type does the same for output.weight, and --leave-output-tensor leaves it unquantized entirely, which the tool’s own README says “increases model size but may also increase quality”.

What an importance matrix does and does not change

An importance matrix — the --imatrix flag — is a set of per-channel activation statistics gathered by running a calibration corpus through the unquantized model. The quantizer uses it to decide where within a block to spend its limited precision, favouring channels that carry more signal.

It is a quality mechanism, not a size mechanism. For a given quant type the bit budget is fixed by the format, so an imatrix-quantized Q4_K_M and a plain Q4_K_M are close to the same size. What it changes is which weights end up accurate, and for the IQ-family types below about three bits it is not optional at all.

Where it does show up in the file size is indirectly: a producer who bothers to build an imatrix is usually a producer with opinions about the embedding and output tensors too, so imatrix files and non-default tensor types tend to travel together. And because the calibration corpus is a choice, two imatrix files from two producers are not interchangeable in quality even at identical size. What a quantized release should disclose covers what to look for.

The remaining causes

• The source precision. Converting from bf16 versus fp16 versus a checkpoint that was already quantized once produces different results. --allow-requantize permits the last of those and warns in its own documentation that it “can severely reduce quality compared to quantizing from 16bit or 32bit”. A file produced that way can be identical in size and materially worse.
• Sharding. --keep-split emits the quantized model in the same shards as the input rather than as one file. A single 4.9 GB file and five 1 GB files are the same model, and a naive size comparison of one shard against a whole file is not measuring anything.
• Per-tensor overrides. --tensor-type accepts regular expressions and can be given several times, so a producer can set any subset of tensors to any type. There is no way to infer this from the filename.
• Metadata. Chat templates, tokenizer data and --override-kv entries live in the file. This is kilobytes, not hundreds of megabytes, but it is why two byte-identical-in-tensors files can still have different checksums.

Finding out what you actually downloaded

The file describes itself. Dump its metadata and its tensor table and the differences stop being mysterious:

# Ships with llama.cpp
python3 gguf-py/scripts/gguf_dump.py model.gguf | head -n 60

# What to look for:
#   general.file_type          — the nominal quant label
#   quantize.imatrix.file      — present only if an imatrix was used
#   quantize.imatrix.dataset   — which corpus calibrated it
#   the tensor table: the type beside token_embd.weight and output.weight

Run that against both files and the several-hundred-megabyte gap resolves into one or two lines: a different type on token_embd.weight, an imatrix key present in one and absent in the other, or a general.quantization_version that reveals two different eras of the tool.

The practical rule this leaves you with: a quant label is a rough indication of size and a poor indication of quality, and two files with the same label from different producers are different artifacts. Pick a producer whose defaults you have looked at, note the file size you expect, verify the checksum so you know you got the file they published, and treat an unexplained size difference as a question about the recipe rather than as evidence of a corrupt download.