AWQ Explained: Protecting the Weights That Matter Most

AWQ and GPTQ produce files of the same size that both say “4-bit” on the tin, and they get there from opposite directions. GPTQ rounds everything and repairs the damage. AWQ decides in advance which weights it refuses to damage, and rescales the matrix so that rounding hurts them less.

The premise: 1% of channels

The AWQ paper (Lin, Tang, Tang, Yang, Dang and Han, presented at MLSys 2024, where it received the Best Paper Award) opens with an observation rather than a method: weights in a large language model are not equally important, and keeping roughly the top 1% of weight channels in FP16 while quantizing the rest to 4-bit recovers most of the quality that uniform 4-bit gives up.

That is an interesting result and a terrible format. A matrix where 1% of the columns are FP16 and 99% are int4 is mixed-precision at the worst possible granularity: the kernel needs two code paths and a gather, the memory layout is ragged, and the speedup that motivated quantizing in the first place evaporates. So the paper’s actual contribution is getting the same protection with a uniformly-4-bit matrix. That is the part worth understanding.

Why the activations pick, not the weights

The obvious way to find important weights is to look at their magnitudes. AWQ does not, and the reason is the same one that makes outlier features a problem in the first place. A weight matters in proportion to the size of the product it contributes to the output, and that is the weight times its input channel. Transformer activations contain a small number of hidden dimensions with magnitudes far outside the rest of the distribution, and a perfectly ordinary-looking weight multiplying one of those dimensions dominates the output.

So AWQ ranks input channels by the average magnitude of the activations flowing through them, collected by running calibration text through the model — not by anything in W. The paper is explicit that selecting by weight magnitude instead performs noticeably worse than selecting by activation magnitude, which is the entire claim in the name.

Scaling instead of mixed precision

Here is the trick. For a per-channel scale s > 1 applied to input channel j, the layer computes the same thing if you multiply that column of the weights by s and divide the corresponding activation channel by s:

y = W · x                      # original
y = (W · diag(s)) · (diag(s)⁻¹ · x)   # identical output

Nothing about the mathematics has changed, but something about the quantization has. Quantization error on a weight is roughly half a grid step, and the grid step is the group’s range divided by the number of levels. Scaling one column up by s while the group’s overall range is set by other, larger columns means that column now occupies more of the available levels — its relative rounding error falls by about s. The paper’s analysis is exactly this: as long as scaling up the salient channel does not push the group’s maximum up much, the salient channel’s error shrinks and everything else’s stays put.

The division on the activation side is not a runtime cost either. In a transformer the linear layer is preceded by a LayerNorm or RMSNorm with its own per-channel weight, or by another linear layer, and diag(s)⁻¹ folds into that operation offline. What ships is a normally quantized 4-bit matrix and a slightly modified norm.

How the scale is chosen

The scale is not free — push s too high and the salient channel becomes the one that sets the group maximum, at which point every other channel in the group loses range. AWQ parameterises the scale by the activation statistics with a single exponent and grid-searches it:

s = mean(|x_j|) ** alpha       # per input channel j
alpha* = argmin over a small grid of alpha in [0, 1] of
         || W·x  -  Q(W·diag(s)) · diag(s)⁻¹·x ||

The objective is the same layer-output reconstruction error GPTQ minimises, evaluated on the calibration batch, and the search space is one scalar per layer over a coarse grid. That is the whole of the optimisation: no gradients, no backward pass, no reconstruction training.

In practice the parameters you set are the ones exposed by Transformers’ AwqConfig and by AutoAWQ: bits (default 4), group_size (default 128, with -1 meaning per-column), and zero_point (default true, i.e. an asymmetric grid). The scale search is internal and has no user-facing knob.

What follows from having no backprop

The absence of a reconstruction-training step is not just a speed property. It changes how the result behaves when your workload differs from the calibration text.

• Less calibration overfitting. The paper’s stated motivation for avoiding backpropagation and reconstruction is precisely that they overfit the calibration distribution; AWQ fits one exponent per layer, which is not enough capacity to memorise a dataset. This is why AWQ is the common choice for multimodal and instruction-tuned models, where a plain-text calibration set is a poor match for the deployment distribution.
• The scale is a property of the channel, not the row. Because the correction lives in a per-input-channel vector, it composes with ordinary per-group weight quantization rather than replacing it. The stored file is the same shape as any other 4-bit grouped format, which is why AWQ checkpoints run on the same kernel families as GPTQ ones.
• It does nothing for activation quantization. AWQ is weight-only — the activations still arrive in FP16 and the outliers in them are untouched. Moving outlier difficulty out of the activations is a different transform with a similar shape; see SmoothQuant, and weight-only quantization for why that division of labour is the default locally.

One consequence catches people out. Because AWQ’s salient-channel selection depends on activation statistics gathered from real forward passes, quantizing a model whose activation distribution you have changed — a heavily fine-tuned checkpoint, a model with a merged LoRA adapter — means re-running the search on that checkpoint. Reusing scales computed for the base model silently protects the wrong channels.