Measuring Quantization Quality With llama.cpp's Perplexity Tool

Perplexity is the cheapest honest answer to “did this quant hurt my model”. It is also easy to quote wrongly, because the number means nothing on its own — only the difference between two runs that were identical in every other respect.

What the number is

llama-perplexity runs the model over a fixed text and reports the exponentiated average negative log-likelihood of the tokens it saw: loosely, how surprised the model was, so lower is better. llama.cpp is explicit about the limits in its own documentation. The value is not comparable across models with different tokenizers, because the tokens being predicted are different objects. Fine-tunes typically score worse while being rated better by humans. And llama.cpp’s numbers are not comparable with other projects’ because the value depends on implementation details.

What it is good for is the one comparison it was adopted for inside the project: the same model, the same corpus, the same settings, at two quantisation levels. That difference is real, it is small, and it is measurable — which is exactly why it needs the uncertainty printed alongside it.

It is also worth being clear about what perplexity does not see. It scores next-token prediction over prose. It says nothing about whether the quantised model still follows an instruction, still emits valid JSON, still calls a tool with the right arguments, or still refuses what it refused before — all of which are the things a quant most visibly breaks. Perplexity is the screening test: cheap, sensitive to gross damage, and no substitute for running your own task once the screen comes back clean.

Getting a corpus

The project’s convention is the WikiText-2 test set, and the repository ships a script that fetches it. Use it if you want numbers comparable with the project’s published table; use your own text if you want to know what the quant does to your workload, which is usually the more useful question. Either way the file must be plain text and must be identical between the two runs.

# the project's corpus
sh scripts/get-wikitext-2.sh
# prints: llama-perplexity -m model.gguf -f wikitext-2-raw/wiki.test.raw

# or your own: concatenate a few megabytes of representative text
cat corpus/*.md > mine.txt

A few hundred kilobytes is enough to get a usable uncertainty; a few megabytes is better and costs a long prefill. Do not use text the model was fine-tuned on, and do not use text you generated with the model — both make the score better in a way that has nothing to do with quantisation.

Running it on two quants

1. Produce the two files you want to compare, if you do not have them. Quantise from the same source with llama-quantize, so that nothing but the quant type differs.
2. Run the baseline. Fix the context length explicitly — the score depends on it, so a run at a different -c is a different measurement, not a noisier one:

   llama-perplexity -m models/model-Q8_0.gguf -f mine.txt -c 512 -ngl all

3. Run the candidate with every flag identical but the model path:

   llama-perplexity -m models/model-Q4_K_M.gguf -f mine.txt -c 512 -ngl all

4. Record both final lines together with the context length, the corpus and the llama.cpp revision. Without those three the pair of numbers is not reproducible, including by you next month.

Reading the output

At start-up the tool prints what it is about to do, in the form calculating perplexity over N chunks, n_ctx=512, batch_size=2048, n_seq=4. The chunk count is the corpus length divided by the context length, which is the mechanical reason a different -c gives a different score: the model gets a different amount of preceding context to predict each token from. It then prints a running estimate per chunk, and finishes with a line of the form Final estimate: PPL = 6.4071 +/- 0.03912.

The +/- is not decoration. llama.cpp derives it by assuming a Gaussian distribution of the correct logits and propagating the error, and on a modest corpus it is frequently larger than the difference between two adjacent quant levels. If your two runs are 6.41 ± 0.04 and 6.39 ± 0.04, you have not shown that one is better. You have shown that the difference is below what this corpus can resolve — which is itself a useful result, and a more honest one than picking the lower number.

For a sense of the scale of the effect you are looking for, the llama.cpp repository publishes its own scoreboard for LLaMA 3 8B in tools/perplexity/README.md, measured on the CUDA backend at the revision stated there. In that table f16 scores 6.233160 ± 0.037828 and Q4_K_M without an importance matrix scores 6.407115 ± 0.039119 — a gap of about 0.175, which is roughly four times the quoted uncertainty on either. Those are the project’s measurements on their hardware and corpus, not predictions for yours; the reason to know them is to recognise a gap of the wrong order of magnitude when your own run produces one.

When perplexity is not sensitive enough

Perplexity averages over every token, so a quant that is fine almost everywhere and badly wrong on a few rare tokens can score well. The tool’s answer to that is KL divergence against the full-precision model’s own logits, which compares the two distributions token by token instead of scoring each against the ground truth.

The procedure is two-pass: run the f16 model with --kl-divergence-base FILE to record its logits, then run the quantised model against that file with --kl-divergence. Budget the disk first — llama.cpp warns that the logit file is very large, giving 11 GiB for LLaMA 2 and 37 GiB for LLaMA 3 on the WikiText-2 test set. What you get back is worth the space: mean KL divergence, the mean and percentiles of the change in the correct token’s probability, and the share of tokens where both models agreed on the top choice. The percentile spread is the interesting one, because a symmetric spread means the quantisation added noise, while a skew towards negative changes means it removed capability.