Which Quantisation Should You Download?

Half of this question has an exact answer and the other half does not. What a quantisation level costs in memory is arithmetic. What it costs in quality depends on your task, and anyone who tells you otherwise is quoting a benchmark run on someone else’s work.

What the names mean

Quantisation stores each weight in fewer bits than it was trained with. The naming in the GGUF world decomposes as follows, and once you can read it you no longer need a table of which files exist:

• The number is the nominal bits per weight. Q4 is about four, Q8 about eight.
• The _K marks a K-quant: weights are stored in blocks with per-block scaling constants, which recover much of the accuracy a flat rounding would lose. The scales are why real bits per weight run above the nominal figure.
• The _S, _M, _L suffix is the mix. Not every tensor gets the same treatment: attention and embedding tensors are commonly kept at higher precision than the feed-forward bulk, and the suffix says how generous that mix is.
• _0 and _1 are the older, simpler block formats. Still present, generally superseded by the K-quants at the same size.
• I-quants (IQ2, IQ3 and friends) use a learned codebook to push below three bits with less damage than the equivalent K-quant. They cost more compute per token to unpack, which on a bandwidth-bound machine is often free and on a slow CPU is not.

Outside GGUF the same idea appears under different names — AWQ and GPTQ produce four-bit GPU-native weights for batching servers, and FP8 is increasingly used where the hardware supports it natively. The trade-offs rhyme; the file formats do not interoperate.

The memory side, exactly

Weights occupy parameters times bits over eight. Applied to a 13B-parameter model, using the effective bits per weight rather than the nominal:

13e9 params

  fp16    16.0 bpw -> 13e9 * 16.0 / 8 = 26.0 GB   (24.2 GiB)
  Q8_0     8.5 bpw -> 13e9 *  8.5 / 8 = 13.8 GB   (12.9 GiB)
  Q6_K     6.6 bpw ->                   10.7 GB   (10.0 GiB)
  Q5_K_M   5.7 bpw ->                    9.3 GB    (8.6 GiB)
  Q4_K_M   4.8 bpw ->                    7.8 GB    (7.3 GiB)
  Q3_K_M   3.9 bpw ->                    6.3 GB    (5.9 GiB)
  Q2_K     3.35 bpw ->                   5.4 GB    (5.1 GiB)

Two things to notice. The interesting territory is narrow: going from Q8 to Q4 roughly halves the file, but going from Q4 to Q3 saves under two gigabytes while asking for a real quality concession. And the steps stop paying at the bottom — Q3 to Q2 saves less than a gigabyte on this model, for the largest quality drop on the list.

There is a second, less obvious benefit. Generation on most hardware is bound by memory bandwidth, and a smaller file means fewer bytes streamed per token. Halving the weights tends to move generation speed in roughly the same direction, which is why a four-bit model can be both the one that fits and the one that feels quick.

How quality actually degrades

The mechanism is worth understanding because it predicts where you will see damage, which is more useful than a single score. Rounding weights perturbs every logit slightly. Where the model was confident, the ranking survives and the output is unchanged. Where two continuations were nearly tied, the perturbation can flip which one wins.

So degradation is not uniform. It concentrates in exactly the places where the model was already near its limit:

• Long chains of dependent steps. Arithmetic, multi-step reasoning and code that must be internally consistent across many lines — a flip early on is amplified rather than corrected.
• Strict formats. JSON, function-call arguments, structured schemas. A single wrong token invalidates the whole output, so per-token error rate translates directly into failure rate.
• Rare knowledge and rare languages. Low-frequency facts were encoded with small margins in the first place.
• Long contexts. Errors accumulate over more tokens, and attention over a large cache is where small perturbations have the most room to compound.

Conversely, summarising, rewriting, classification, extraction and ordinary conversation are strikingly robust — these are tasks where many continuations are acceptable and the confident ranking is rarely close.

One more rule that holds up: a bigger model at a lower quant usually beats a smaller model at a higher quant, for the same file size — until you go very low, at which point the larger model collapses faster than the smaller one would have. Somewhere below about three bits the trade reverses. Where exactly depends on the model, which is why the next section exists.

An acceptance test you can run

This takes twenty minutes and replaces every published quality table with evidence about your workload. The design is paired: same prompts, same seed, two quant levels, difference judged by you.

# 1. take 30-50 real prompts from your own logs, including the
#    hardest ones and any that must produce valid JSON.

# 2. run each against two quants, temperature 0, same seed,
#    identical chat template.
for q in Q8_0 Q4_K_M; do
  llama-server -m ./m-$q.gguf -c 8192 -ngl 99 --port 8080 &
  sleep 20
  while read -r p; do
    curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
      -d "{\"model\":\"m\",\"temperature\":0,\"seed\":1,
           \"messages\":[{\"role\":\"user\",\"content\":$p}]}" \
      | jq -r '.choices[0].message.content' >> out-$q.txt
  done < prompts.jsonl
  kill %1
done

# 3. mechanical checks first - these are pass/fail, not opinion:
#    does every JSON output parse? does every code output compile?
#    is every required field present?

# 4. only then read the pairs side by side for the prompts that passed.

Step three is the one that earns its keep. A drop in structured-output validity is measurable without judgement, it is the failure mode that most often makes a quantised model unusable in an application, and it is invisible in a chat window where you would have shrugged at slightly worse prose.

Perplexity is available as a cheaper proxy — llama-perplexity against a held-out text file will give you a number per quant — but treat it as a smoke alarm rather than a verdict. It detects gross damage reliably and predicts task performance only loosely.

Quantising the cache instead

If you are a gigabyte or two short, the KV cache is often the cheaper thing to compress. An eight-bit cache halves that term, and because keys and values are transient rather than the model’s learned structure, the quality cost is typically smaller than dropping a weight quant level:

llama-server -m m-Q4_K_M.gguf -c 32768 -ngl 99 \
  --cache-type-k q8_0 --cache-type-v q8_0
# on the 8B example: 4.0 GiB of fp16 cache at 32k becomes 2.0 GiB

Quantising the key cache is generally safer than the value cache; if you see degradation, relax the value cache first. Run the same acceptance test — the point of having a test is that you stop guessing about any of these.

A defensible default

Start at Q4_K_M for the largest model that fits with your real context length. It is where the file-size curve stops being generous and before the quality curve starts being cruel. Move up to Q5_K_M or Q6_K if you have headroom and your acceptance test shows a difference. Go below four bits only when the alternative is a materially smaller model, and only after testing, because that is the region where the answer genuinely varies between models.