What Happens When You Quantize an Already-Quantized Model

You have a Q4_K_M file, you want a smaller one, and llama-quantize stops before writing a byte. The refusal is deliberate and the message names the reason precisely.

The error

llama_model_quantize: failed to quantize: requantizing from type q4_K is disabled
main: failed to quantize model from 'Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf'

The type name varies with what you fed it — q4_K, q5_K, q8_0, q6_K — but the sentence is fixed. It comes from llama_model_quantize_impl in src/llama-quant.cpp, at the point where the tool is about to produce float data for a tensor. If the source tensor is already quantized and allow_requantize is false, it throws instead of dequantizing. The default for that parameter is false.

The immediate fix, which you should read the next two sections before using, is the flag it is named after:

llama-quantize --allow-requantize \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  Meta-Llama-3.1-8B-Instruct-Q3_K_M.gguf Q3_K_M

The tool’s own help text is unusually direct about it: “allow requantizing tensors that have already been quantized”, followed by a warning that this can severely reduce quality compared to quantizing from 16-bit or 32-bit. That warning is the whole content of this page.

Why the guard exists

Quantization maps each weight to the nearest point on a grid whose spacing is set by a scale computed for a small block. Doing it once costs you the distance from the true weight to the nearest grid point, and that error is bounded by half a grid step and roughly uniform across the block.

Requantizing does not start from the true weight. It starts from the grid point, because that is all that survived. The tool dequantizes the Q4_K tensor back to float — recovering 16 distinct values per block arranged on a lattice, not the original continuous distribution — and then fits a new, coarser grid to that. The second fit is solving a different problem from the one it was designed for.

Two consequences follow, and they are why the guard is a refusal rather than a warning:

• The errors add rather than replacing each other. Your final weight is off by the first rounding plus the second, and the second cannot correct the first because the information needed to do so is gone.
• Scale selection is fitted to the wrong distribution. A quantizer chooses each block’s scale to minimise error against the weights it sees. What it sees on a requantization pass is a lattice with gaps, so the scale it picks is optimal for the lattice and not for the weights the lattice was standing in for. The outlier handling that k-quants exist to do has already been applied once, to data that is no longer there.

The k-quant mixtures make this worse in a way that is easy to miss. As the difference between Q4_K_M and Q4_K_S shows, a Q4_K_M file already stores certain tensors at Q6_K because the quantizer judged them sensitive. Requantizing to Q3_K_M re-derives that judgement from scratch and will happily round those same tensors hard, having lost the precision that was deliberately spent on them.

What the override actually does

Nothing dramatic and nothing protective. Setting --allow-requantize flips one boolean; the dequantize-then-fit path runs, the file is written, and it loads. There is no warning at load time and nothing in the resulting GGUF header records that its weights went through two quantizers. The output is a normal-looking file whose quality is worse than a file of the same name produced from the original weights, by an amount nobody can tell you in advance because it depends on both levels and on the model.

The size saving is real, which is what makes the trade tempting. Going from Q4_K_M to Q3_K_M on an 8B saves roughly a gigabyte. But you are spending accumulated precision to buy it, and the tasks that notice first are exactly the ones described in choosing a quantization level by use case — code and structured output, where a flipped token is a failure rather than a variation.

What to do instead

Quantize from the original weights. The path is:

1. Fetch the original repository rather than a GGUF one — hf download <org>/<model>. This is the large download, typically two bytes per parameter for a BF16 checkpoint. Some repositories are gated and require accepting a licence on the model page first; that is a condition to satisfy, not an obstacle to work around.
2. Convert to an unquantized GGUF: python convert_hf_to_gguf.py <model-dir> --outtype bf16 --outfile model-bf16.gguf. Nothing is lost here; it is a format change.
3. Quantize once, to the level you actually want: llama-quantize model-bf16.gguf model-Q3_K_M.gguf Q3_K_M. If you want several levels, run this once per level from the same bf16 file rather than chaining them.

If you would rather not download the full checkpoint, the better move is usually to find someone who already published the level you want. Most popular models have a full ladder of quants uploaded, and reading the filenames is faster than re-deriving anything. Downloading a 4 GB file you want is cheaper in every dimension than degrading a 5 GB file you have.

When requantizing is defensible

There are narrow cases where it is the right call, and it is worth being precise about them rather than treating the flag as forbidden:

• Starting from Q8_0. At 8.5 bits per weight the grid is fine enough that the first rounding is small relative to the second. Requantizing Q8_0 to Q4_K_M is a meaningfully different proposition from requantizing Q4_K_M to Q3_K_M, and if a Q8_0 file is what you have and the original is a 100 GB download you cannot make, this is a reasonable trade.
• The original weights no longer exist. A merged model or a fine-tune published only as a quantized GGUF leaves you no alternative. Use --leave-output-tensor, whose help text notes it increases size but may increase quality “especially when requantizing” — the output tensor is where a second rounding is most visible.
• You are going to evaluate the result anyway. If you have a task-specific harness, requantize, measure, and decide from the number. The objection is to doing it blind, not to doing it.

One last thing worth checking before any of this: confirm what you actually have. gguf_dump.py --no-tensors reports general.file_type, and a file whose name says Q4_K_M but whose header says something else is a real possibility given that filenames are written by people. If the input turns out to be F16 already, the error you started with was never about requantizing at all.