Mixed-Precision Quantization: Keeping Some Layers at Higher Bits

Nothing about a transformer says every layer deserves the same number of bits. Some matrices tolerate 3-bit rounding almost invisibly and some fall apart, and once you accept that, a uniform bake starts looking like what it is: a budget spent evenly because spending it unevenly requires knowing something.

Uniform bit width is a strange default

A 4-bit model is a model in which the attention output projection, the MLP down projection and the query projection are all given exactly sixteen levels, despite differing by an order of magnitude in how much the layer’s output error propagates into the next block. The reason it is the default is not that it is optimal. It is that it needs no information: uniform quantization is what you do when you have not measured anything.

The moment you do have a sensitivity ranking, the question changes from “how many bits” to “how should this many bits be distributed” — a constrained allocation problem with a fixed budget and a per-layer cost curve. Every serious format now solves some version of it, which is why bits-per-weight figures in this territory are almost never integers.

Three ways to rank sensitivity

• By hand, from architecture. llama.cpp’s K-quant mixes promote a fixed list — attention.wv, feed_forward.w2, and at lower base widths attention.wo — and all variants special-case the output projection. No measurement per model; a rule derived once from where magnitude concentrates and applied everywhere. Cheap, robust, and necessarily approximate. The exact lists are in what K, M and S mean in a GGUF quant name.
• By measurement, per layer. ExLlamaV2 quantizes each linear layer at several candidate settings, records the reconstruction error each produces against calibration data, and solves the allocation against those curves. This is the most informative ranking available and it costs a full measurement pass; EXL2 covers the mechanics.
• By activation statistics, within a layer. AWQ is mixed-precision in effect without being mixed-precision in storage: it identifies roughly the top 1% of channels by activation magnitude and protects them with a scaling transform rather than with extra bits. Same objective, achieved without a ragged memory layout.

The three differ in granularity as much as in method — per-tensor-type, per-layer, per-channel — and the granularity is what determines whether a kernel can consume the result at speed.

The arithmetic of a non-uniform bake

The question worth answering is what a promotion actually costs, and the answer depends entirely on how much of the model the promoted tensors represent. Take a 13B-class model, 40 layers, hidden 5120, intermediate 13824, and suppose you promote the down projection from 4 bits to 6.

per-layer linear parameters
  q,k,v,o   5120x5120 x4      = 104.9M
  gate,up   13824x5120 x2     = 141.6M
  down      5120x13824        =  70.8M
  total                       = 317.3M

down_proj share = 70.8 / 317.3 = 22.3%

uniform 4-bit    : 4.000 bits per weight
promote down to 6: 0.777*4 + 0.223*6 = 4.446 bpw

size for 12.7e9 linear params
  uniform  12.7e9 * 4.000 / 8 = 6.35 GB
  promoted 12.7e9 * 4.446 / 8 = 7.06 GB
  cost of the promotion       = 0.71 GB  (+11%)

Now run it the other way, which is the more useful direction. If the budget is fixed at 4.446 bpw, the alternative to promoting the down projection is spending the same bytes uniformly — and 4.446 bits is not a representable uniform width, so in practice the comparison is against 5-bit uniform at 5.0 bpw, which costs 7.94 GB. The non-uniform bake is 0.88 GB smaller than the nearest uniform option above it, while putting all of the extra precision on the tensor the ranking said needed it.

That is the entire value proposition, and it is a function of one number: the promoted tensors’ share of the parameters. Promoting something that is 2% of the model is nearly free and buys little unless that 2% is genuinely critical. Promoting something that is 45% of the model — the gate and up projections together, on this architecture — is barely different from raising the base width.

What actually ships

Three exceptions recur across otherwise unrelated formats, and their agreement is the strongest available evidence that the ranking is real rather than a preference.

• The output projection is always promoted or skipped. The k-quants pull request states that all quantization variants use 6-bit quantization for output.weight, later refined to Q5_K by convention for the QX_K variants; GPTQ and AWQ recipes almost universally exclude lm_head. This layer maps hidden states to vocabulary logits with nothing downstream to absorb its errors.
• One-dimensional tensors are never quantized. Norm weights and biases stay in floating point in llama.cpp and in every other format in common use. They are a negligible share of the parameters and a large share of the numerical sensitivity — the cheapest exception available.
• The down projection and value projection recur. Both llama.cpp’s hand-picked mixes and measurement-driven allocators tend to spend on the same places, which is what you would expect if the ranking reflects something structural about where magnitude concentrates.

What it costs you

• Kernel fragmentation. A model with three quantization types in it needs a kernel for each, and a runtime that supports one but not another will fall back to a slow path for part of the model. This is the main reason mixed-precision lives inside formats — GGUF, EXL2 — rather than being something users assemble.
• The allocation is calibration-dependent. A measurement-driven ranking inherits every property of the corpus it was measured on. A model whose sensitivity was ranked on English prose has spent its bits where English prose needed them; see what a calibration dataset actually does.
• It competes with group size for the same budget. Going from group 128 to group 32 costs about 0.47 bits per weight uniformly. Promoting a 22% tensor by two bits costs 0.45. These are two ways to spend the same half-bit and they help in different situations — group size targets weights near an outlier, promotion targets whole tensors whose errors propagate. Neither dominates.