Choosing a Quantization Level by Use Case

The same quantization that leaves a chat model conversational will quietly break a model writing JSON, and the reason is not that code is “harder”. It is that code has no acceptable second choice at most positions and prose has thousands.

What quantization actually perturbs

Quantization replaces each weight with the nearest value on a coarse grid, storing a scale per small block so the grid can adapt locally. The error at any one weight is tiny and roughly random. What it changes at the output is not the model’s knowledge but the logits — the scores over the vocabulary — which shift by small amounts in unpredictable directions at every position.

A shifted logit only matters if it changes which token gets selected. At the vast majority of positions it does not, because the leading token leads by a wide margin. The whole question of what quantization costs is therefore the question of how often your task puts the model at a position where the margin is small — and different tasks put it there at very different rates.

There is a second, sharper effect worth naming. Transformer activations contain a small number of outlier features with magnitudes far above the rest, and a uniform grid fitted to a block containing one of them spends most of its resolution representing the outlier. This is exactly what the k-quant mixtures respond to by promoting particular tensors to higher precision, and it is why the M and S variants of one level differ by a tensor list rather than a bit count.

Low-margin positions flip first

Consider two positions in a generation. In the middle of an explanatory sentence, dozens of continuations are fluent and appropriate; the top few tokens have similar scores, and if a small perturbation swaps the first and third, the sentence is still fine. The output is different, and it is not worse.

Now consider the position immediately after a closing brace in generated JSON, or after a variable name that must match a declaration forty lines up. There is exactly one correct token. The model may know it with a large margin, in which case quantization changes nothing — or with a small one, because the reference is distant and the evidence is weak, in which case a perturbation that would have been invisible in prose selects the wrong identifier.

The asymmetry is not in the model’s difficulty. It is in what happens after the flip. Prose degrades gracefully because the space of acceptable outputs is enormous. Code and structured output degrade catastrophically at the token level because the space of acceptable outputs is a single point, and everything else is a syntax error or a reference to something that does not exist.

Why exact-match tasks suffer most

Grouping tasks by how large their set of acceptable outputs is turns out to predict quantization tolerance better than any notion of difficulty:

• Open-ended chat, summarisation, brainstorming. Very large acceptable set. Most flipped tokens produce a different acceptable output. These tolerate aggressive quantization best.
• Classification and extraction with a short label set. Small output, and the decisive token is usually decided by a wide margin. Robust in practice, and cheap to verify because you can check the label is in the allowed set.
• Code. Long outputs where identifiers, brackets and indentation must be exactly right and depend on distant context. One wrong token is a compile error, not a stylistic difference.
• Structured output — JSON, SQL, tool call arguments. The strictest case. Field names must match a schema exactly, types must be right, and a single stray character makes the whole payload unparseable, so the effective error rate is per-document rather than per-token.
• Long-context retrieval and citation. Requires copying spans accurately from far back in the context, where the attention signal is weakest and therefore the margins are smallest.

One important qualifier on structured output. If you are using grammar-constrained decoding — llama.cpp’s GBNF grammars, or a JSON-schema-constrained response format — the sampler masks every token that would be invalid, so purely syntactic breakage is eliminated regardless of quantization level. What constraint cannot fix is semantic error: the schema-valid object with the wrong field populated. Constrained decoding moves quantization damage from “unparseable” to “parseable and wrong”, which is harder to notice, not easier.

Length turns a small rate into a large one

The last piece is why a per-token error rate that sounds negligible is not. If a quantization level flips a decisive token with probability p at each of n critical positions, the chance the whole output is correct is roughly (1 - p)^n. A 400-token function with 40 positions where exactness matters, at a flip rate of one in a hundred, succeeds about 67% of the time. The same rate over a three-token classification label is invisible.

That is a shape argument, not a measurement — nobody has published p for your model at your quant level, and it varies by model, level and prompt. The shape is what matters: the penalty grows with the number of positions that must be exactly right, so long code generation is the worst case and short constrained answers are the best, from exactly the same model at exactly the same bits.

Choosing, and checking your choice

Given the mechanism, a reasonable default ordering is: take the highest level that leaves room for the context you need, and prefer spending your memory budget on precision rather than parameters when the task is exact-match. A smaller model at a higher quant level frequently beats a larger model at an aggressive one for code, because the larger model’s advantage is knowledge and the failure you are fighting is precision.

The check is cheap and specific to you. Build twenty inputs from your real workload with known-correct outputs, run them at two levels with temperature at zero, and count exact matches rather than reading the outputs for vibes. For structured output, count parse failures and schema violations separately from semantic errors; they have different causes and only one of them is fixed by constrained decoding. That harness takes an afternoon and answers the question for your task, which no published table can.

Two things to hold on to. Memory saved by dropping a level is available for context, and running out of context is a total failure while a slightly noisier logit is a partial one. And whatever you choose, record which file you tested, because the same quant name from a different uploader is not necessarily the same file.