SmoothQuant Explained

Weights quantize to int8 easily. Activations do not, and the whole difficulty of W8A8 inference is contained in that asymmetry. SmoothQuant’s answer is not a better activation quantizer — it is a rescaling that makes the activations someone else’s problem, offline, for free.

The problem: activations quantize badly

Weight distributions in a trained transformer are well-behaved: roughly normal, flat across channels, no channel dramatically different from its neighbours. Activations are not. From about 6.7B parameters onward, transformer activations contain systematic outlier features — a small number of hidden dimensions whose magnitudes sit far outside the rest, consistently, across layers and across most tokens. The evidence for this is in Dettmers, Lewis, Belkada and Zettlemoyer’s LLM.int8() paper (NeurIPS 2022), and the mechanism is worked through in outlier features and why LLM quantization breaks without handling them.

The interaction with granularity is what makes this fatal. Activations are quantized per-tensor or per-token, never per-channel along the reduction dimension, because a scale that varies along that axis cannot be pulled out of the accumulation. So the outlier channel and every ordinary channel share one scale, and that scale is set by the outlier. With 256 levels and a range dictated by a value twenty times the typical magnitude, the ordinary channels are left using a dozen of those levels. That is effectively 3- or 4-bit resolution on the values that carry most of the signal.

The transform

SmoothQuant (Xiao, Lin, Seznec, Wu, Demouth and Han, ICML 2023) observes that the difficulty is unevenly distributed between the two operands and can be redistributed. A linear layer computes Y = XW. Insert a diagonal matrix and its inverse between them:

Y = X W
  = (X · diag(s)⁻¹) · (diag(s) · W)
  =        X̂        ·       Ŵ

The product is unchanged — this is an algebraic identity, not an approximation. But X̂ and Ŵ are different matrices from X and W, and if s_j is large for the outlier channel j, then that channel of the activations has been divided down towards the rest of the distribution while the corresponding row of the weights has been multiplied up.

The activation matrix now has a much narrower per-tensor range, so its int8 grid is far finer for the channels that matter. The weight matrix has one row that is larger than it was — but weights are quantized per-output-channel, so that row gets its own scale and absorbs the increase without affecting any other row. Difficulty has moved from an operand that could not isolate it to one that can.

Alpha: how much difficulty to move

Move too much and you have simply broken the weights instead. The paper parameterises the trade with a single exponent it calls the migration strength:

s_j = max(|X_j|)^alpha  /  max(|W_j|)^(1 - alpha)

alpha = 0    →  s_j = 1 / max(|W_j|)   : all difficulty on activations
alpha = 1    →  s_j = max(|X_j|)       : all difficulty on weights
alpha = 0.5  →  geometric balance between the two

The paper’s default is alpha = 0.5, described as a well-balanced point that splits the difficulty evenly, and it is what they use for OPT and BLOOM. GLM-130B needed alpha = 0.75, because that model has roughly 30% outliers — far more than the others — and therefore needs more of the difficulty pushed onto the weight side before the activations become tractable.

The value is model-specific in practice, and the reference implementation’s own results table lists different alphas per model family: 0.85 for Llama-2 and Llama-3-70B, 0.8 for Mistral-7B and Mixtral-8x7B, 0.6 for Falcon-7B, 0.7 for Falcon-40B. Read that spread as the signal it is — alpha is a per-model tuning parameter, and copying one model’s value to another is a guess.

Why the rescaling is free at runtime

Dividing every activation by a per-channel vector at inference time would be an extra elementwise pass over a large tensor at every layer, which would eat the speedup. It does not happen, because in a transformer the thing immediately preceding a linear layer is almost always a LayerNorm or RMSNorm with its own learned per-channel gain.

Folding diag(s)⁻¹ into that gain is a one-line offline operation: divide the norm’s weight vector elementwise by s. The norm then emits already-smoothed activations, and the runtime does exactly the work it did before. On the weight side, diag(s) · W is baked into the checkpoint. Nothing at inference knows SmoothQuant happened.

The catch is structural: the fold requires something with a per-channel scale in front of the linear layer. Where the preceding operation cannot absorb the vector — a residual branch feeding several consumers with different smoothing needs, a fused QKV projection whose input is shared — the transform either has to be applied to all consumers consistently or not at all. This is why SmoothQuant implementations are architecture-aware rather than a generic wrapper.

Where it fits, and where it does not

SmoothQuant exists to make W8A8 work. That is a different goal from the 4-bit weight formats that dominate local inference, and the two are not competitors so much as answers to different constraints.

• It buys compute, not just memory. Because both operands are int8, the multiply itself runs on integer tensor cores at roughly double the FP16 throughput. Weight-only 4-bit formats leave the multiply in FP16 and save only bandwidth — see weight-only quantization for why that is the right trade at small batch sizes and the wrong one at large.
• It is a serving technique. The advantage shows up in the compute-bound regime: large batches, long prefills, high throughput. At batch size one it does relatively little, because the multiply was not the bottleneck.
• The efficiency levels are a real choice. The paper defines O1 through O3, differing in how the activations are quantized: per-token dynamic, per-tensor dynamic, and per-tensor static, in increasing order of aggression and efficiency. The static variant depends on calibrated ranges and is the one that clips if a real input exceeds them; see dynamic quantization.
• 8 bits is the target, not 4. Nothing about the smoothing transform prevents combining it with 4-bit weights, and successors do, but SmoothQuant as published is a W8A8 method and claims W8A8 results.