What K, M and S Mean in a GGUF Quant Name

Q4_K_M is three pieces of information joined by underscores, and every one of them is defined in llama.cpp’s source rather than being a vibe. The last letter in particular is not “medium quality” — it is a specific list of tensors that get more bits than the rest.

Reading the name

• Q4 — the base bit width for most weights. Q2 through Q8 exist.
• K — the quantization family. K means “K-quant”, the super-block scheme introduced to llama.cpp in 2023, as opposed to the older Q4_0 and Q4_1 formats which have no letter and use a simpler block layout.
• M — the mix. S, M and L select which tensors are promoted above the base width.

So Q4_K_M is: K-quant family, 4 bits as the base, medium mix. Q4_0 is the legacy format with none of this. Names like IQ4_XS belong to a third family — the I-quants — which use importance-matrix-guided codebook lookup rather than the K-quant block structure, and are a different design.

What the K actually is

A K-quant stores weights in blocks, and stores the blocks’ own scales in a quantized form inside a larger super-block that carries a floating-point scale over them. It is a two-level hierarchy, and it exists so that scales can be fine-grained without costing a full FP16 per block.

The structures, as documented in llama.cpp’s k-quants pull request:

Q2_K  super-block of 16 blocks x 16 weights
      block scales and mins quantized to 4 bits
      → 2.5625 bits per weight

Q3_K  super-block of 16 blocks x 16 weights
      scales quantized to 6 bits
      → 3.4375 bits per weight

Q4_K  super-block of 8 blocks x 32 weights
      scales and mins quantized to 6 bits
      → 4.5 bits per weight

Q5_K  same super-block structure as Q4_K
      → 5.5 bits per weight

Q6_K  super-block of 16 blocks x 16 weights
      scales quantized to 8 bits
      → 6.5625 bits per weight

The half-bit overheads are the scale hierarchy. Q4_K’s 4.5 bits per weight is 4 bits of payload plus 0.5 bits of block scales and mins — a 12.5% overhead, which is what buys a scale every 32 weights rather than every row. That is the same trade computed generally in quantization granularity, at the fine end of the range.

“Type-0” and “type-1” in the source refer to symmetric versus asymmetric grids: type-0 stores a scale only, type-1 stores a scale and a minimum. Q4_K and Q5_K are type-1, which is why their descriptions mention mins and Q3_K’s and Q6_K’s do not.

S, M and L are tensor lists

Here is the part that is genuinely undocumented outside the source. The suffix does not scale anything uniformly; it names specific tensors that get a different quantization type. From the same pull request:

Q2_K     Q4_K for attention.wv and feed_forward.w2
         Q2_K everywhere else

Q3_K_S   Q3_K for everything
Q3_K_M   Q4_K for attention.wv, attention.wo, feed_forward.w2
         Q3_K everywhere else
Q3_K_L   Q5_K for attention.wv, attention.wo, feed_forward.w2
         Q3_K everywhere else

Q4_K_S   Q4_K for everything
Q4_K_M   Q6_K for HALF of attention.wv and feed_forward.w2
         Q4_K everywhere else

Q5_K_S   Q5_K for everything
Q5_K_M   Q6_K for HALF of attention.wv and feed_forward.w2
         Q5_K everywhere else

Three observations that make the pattern legible. First, S is always the pure case — one type throughout — so Q4_K_S is genuinely uniform 4-bit K-quant and nothing else. Second, the promoted tensors are always the same two or three: attention.wv (the value projection), feed_forward.w2 (the down projection), and at the 3-bit level attention.wo as well. Third, the “half” in the M variants is literal — half the layers get the promotion, not half of each tensor.

Why those tensors? Both sit downstream of an operation that concentrates magnitude — w2 takes the output of the gated activation, wv feeds the attention-weighted sum — and both are places where quantization error propagates rather than averaging out. The mixes are a hand-picked sensitivity ranking, which is the same idea mixed-precision quantization solves for automatically and EXL2 solves by measurement.

Two categories of tensor sit outside the suffix system entirely, and they explain part of why a file is larger than the bits-per-weight arithmetic predicts. One-dimensional tensors — the norm weights and biases — always remain f32 in llama.cpp and are never quantized at all; they are a small share of the parameters and a large share of the numerical sensitivity. And the output projection is special-cased across the board: the k-quants pull request states that all quantization variants use 6-bit quantization for output.weight, noting it lowers Q4_0’s perplexity by about 0.03 at 7B, with a later refinement to Q5_K for the QX_K variants by convention.

That last detail is worth holding on to, because it is the same decision every serious format makes independently. The layer that maps hidden states onto the vocabulary is the one whose errors turn directly into wrong token probabilities, with no subsequent layer to absorb them. EXL2 keeps the head at higher precision; GPTQ and AWQ tooling excludes lm_head from quantization by default in most recipes. Three unrelated projects arriving at the same exception is a reasonable signal that it is structural rather than a preference.

The published bits per weight

You can compute a mix’s effective bits per weight from the table above if you know the model’s tensor shapes, but the shortcut is that S variants land on their base type’s figure and M variants land a little above it. Q4_K_S is essentially 4.5 bpw across the quantized tensors; Q4_K_M is higher by however much of the model those half-promoted wv and w2 tensors represent, which depends on the architecture’s head count and MLP ratio and is not a constant across model families.

For sizing, use the file size the publisher lists rather than a formula. For understanding why two 4-bit files differ in size by 8%, the table above is the answer.

Picking one

Two things follow from the structure rather than from anyone’s benchmark.

• M over S at the same base width is a cheap purchase. You are paying for higher precision on a small, specifically chosen subset of tensors rather than on everything. That is a better use of the same bytes than moving the base width up, which is why the M variants are what most publishers upload.
• A larger model at a lower mix is usually available. Q4_K_M of a 13B and Q6_K of a 7B occupy similar space. Which is better is a question about your task and not answerable from the format — but it is the comparison worth setting up, and it is the one the file-size column hides. The size/quality reasoning behind why 4 bits is where most people land is in why 4-bit became the local-inference default.