Dynamic Quantization for LLM Inference

Dynamic and static quantization differ in exactly one thing: whether the scale for a tensor is computed from that tensor at runtime or fixed in advance from calibration data. Everything else people attribute to the distinction — accuracy, robustness, speed — follows from that single choice.

The difference is when the scale is computed

To quantize a tensor to int8 you need its range. Take the absolute maximum, divide by 127, and you have a scale that maps the tensor onto the integer grid.

Static means that absolute maximum was measured during a calibration pass, over a corpus somebody chose, and stored in the checkpoint. At inference the scale is a constant read from memory.

Dynamic means the absolute maximum is computed from the actual tensor, in this forward pass, for this batch of tokens. It requires a reduction over the tensor before the quantization can happen, and it produces a different scale for every request.

The question only arises for activations. Weights are the same on every forward pass, so there is nothing dynamic to compute — a scale derived from a weight matrix is a constant by construction.

How static scales fail

A static scale is a bet that no future input will exceed the range you measured. When it loses, the values above the range are clipped to the grid’s maximum, and clipping is not a graceful degradation — it replaces a large value with a smaller one, which is a much larger error than rounding.

Transformer activations make this bet worse than it sounds, because they contain systematic outlier features whose magnitudes dwarf the rest of the distribution and whose presence is heavily input-dependent. A calibration corpus of English prose may simply never produce the activation magnitudes that a long code block, a table of numbers, or text in an unrelated script produces. The quantized model then works fine in testing and clips in production, on the inputs least like the calibration set.

Dynamic scales cannot have this failure. The scale is derived from the tensor it is scaling, so by construction nothing in that tensor is out of range. Outliers still cost you resolution — a large maximum makes a coarse grid step for everything else — but they cost resolution rather than causing clipping, and that is a materially different failure. The resolution problem is what SmoothQuant addresses separately.

What dynamic costs per token

The cost is a reduction and a pass. For an activation tensor of shape [tokens, hidden] quantized per-token, computing the scales means one max-reduction along the hidden dimension, then one elementwise multiply-and-round over the whole tensor. Both are memory bound and both touch the tensor once.

per linear layer, per forward pass:
  read   tokens * hidden * 2 bytes   (fp16 activations)
  reduce to tokens scales
  write  tokens * hidden * 1 byte    (int8 activations)

static instead:
  read   tokens * hidden * 2 bytes
  write  tokens * hidden * 1 byte
  (scale is a constant already in registers)

So dynamic adds a reduction over a tensor that had to be read anyway, and in a fused implementation the reduction rides along with a pass that was already happening. That is why dynamic per-token activation quantization is the common default in practice rather than the expensive option — the theoretical overhead is real and the measured gap is usually small enough that the robustness wins.

The overhead grows in relative terms as the layer gets cheaper. On a small batch with short sequences, where the matrix multiply is itself tiny, an extra pass over the activations is a larger fraction of the total, which is one reason static per-tensor quantization survives in latency-critical and edge deployments.

There is a second cost that is not about speed and catches people out in testing. A dynamic per-tensor scale is computed over the whole activation tensor, which includes every sequence in the batch. Two requests batched together therefore share a scale derived partly from each other’s activations, and the same prompt run alone, or batched with a different neighbour, gets a slightly different scale and can produce a different token. The output is still correct; it is just not bitwise reproducible as a function of the prompt alone. Per-token dynamic scaling removes this — each row gets its own scale and batch composition stops mattering — which is a real argument for O1-style per-token quantization beyond accuracy, and a real reason a test suite that pins exact outputs will flake under per-tensor dynamic scaling and not under static.

Why weights are always static

It is worth being explicit about this, because “dynamic quantization” in the older PyTorch sense described quantizing weights ahead of time and activations at runtime, which reads as if the weights were the dynamic part.

A weight matrix is identical on every forward pass. Its absolute maximum, per row or per group, is a fixed number. Computing it at runtime would produce the same answer every time at the cost of a reduction over the largest tensors in the model — strictly worse than storing it. Every weight-quantization format therefore stores its scales, and granularity is the only decision left on that side.

Reading it off a real config

The vocabulary varies by tool, so here is how the distinction usually surfaces:

• SmoothQuant’s O1/O2/O3 levels. O1 is per-token dynamic activations, O2 per-tensor dynamic, O3 per-tensor static — increasing efficiency and decreasing robustness, in that order, and the paper names O3 as the most aggressive.
• W8A8 schemes in serving stacks. A scheme named “W8A8-dynamic” or similar needs no activation calibration at all; a static one requires a calibration pass and produces activation scales in the checkpoint. If a quantization recipe asks for a dataset and you are only quantizing weights, something is quantizing activations statically.
• FP8 schemes. Transformers’ FineGrainedFP8Config documents activation_scheme defaulting to "dynamic", and describes it as the only currently supported value — a useful signal about which way the practice has settled for FP8.
• Weight-only formats have no entry here at all. GPTQ, AWQ, NF4 and the GGUF K-quants leave activations in floating point, so the static/dynamic question does not arise. See weight-only quantization.