Quantization Granularity: Per-Tensor, Per-Channel and Per-Group Scales

Per-tensor, per-channel, per-token and per-group are four names for one question: how many numbers share a scale? Everything else about them — the memory they cost, the accuracy they preserve, whether a kernel can use them — follows from the answer.

One mechanism, three resolutions

A quantized value is stored as a small integer. Recovering the real number it stands for takes a scale, and optionally a zero point: real ≈ scale × (code − zero). The scale has to come from somewhere, and it is always derived from the range of some set of values. Granularity is the size of that set.

• Per-tensor. One scale for the entire matrix. Every weight in a 4096×11008 layer — 45 million of them — is measured against a single number derived from the largest absolute value anywhere in it.
• Per-channel. One scale per row (for weights, usually per output channel), so a matrix with 4096 output channels carries 4096 scales. Each output neuron’s weights are scaled against their own range.
• Per-group. One scale per contiguous run of g weights within a row, along the input dimension. With g = 128 and 11008 input features, that row now carries 86 scales instead of one.

For activations the equivalent axes have different names for the same reason: per-token means one scale per row of the activation matrix, which is one per token in the batch, and it is the activation-side analogue of per-channel.

What each resolution costs

Take one weight matrix of shape [4096, 11008] — 45.1 million weights — at 4 bits, and store a 16-bit scale and 4-bit zero point per scale group. The weight payload is the same in every row of the table below; only the metadata moves.

weights           45.1e6 * 4 bits           = 22.55 MB payload

per-tensor        1 scale                   = 20 bits
                  overhead per weight       ≈ 0.0000004 bits
per-channel (row) 4096 scales               = 81,920 bits
                  overhead per weight       = 20/11008 ≈ 0.0018 bits
per-group g=128   4096 * 86 = 352,256       = 7.05e6 bits
                  overhead per weight       = 20/128  = 0.156 bits
per-group g=32    4096 * 344 = 1.41e6       = 2.82e7 bits
                  overhead per weight       = 20/32   = 0.625 bits

The shape of that result is the thing to carry away. Per-tensor and per-channel are both effectively free — per-channel costs under a twentieth of one percent of the payload — and the first granularity that costs anything measurable is per-group. This is why per-channel weight quantization is universal and never argued about, and why the only granularity anyone tunes is the group size. The arithmetic of that specific choice is in group size in GPTQ and AWQ.

The accuracy side of the same table is more dramatic than the cost side, which is the point. Suppose one weight in that 45-million-weight matrix has magnitude 0.9 and the typical weight has magnitude 0.02. A per-tensor 4-bit grid spans [-0.9, 0.9] in 16 steps, so the step is 0.12 — six times larger than a typical weight. Essentially every weight in the matrix rounds to zero. Per-channel confines that disaster to one output row; per-group with g = 128 confines it to 128 weights out of 45 million. The metadata cost bought three orders of magnitude of containment, which is why nobody ships per-tensor weights below 8 bits.

A note on the earlier arithmetic: the per-channel row assumes grouping along the input dimension, so the 11008 weights of one output channel share 20 bits. If a format instead stores one scale per input channel, the divisor is 4096 and the overhead is 0.0049 bits per weight — the same order, different constant. This is a real ambiguity in the phrase “per-channel”, and it is worth checking which axis a format means before matching a file size against group size in GPTQ and AWQ.

Why weights can afford the fine end

Weights have a property that makes fine granularity cheap in a second sense: they are known before inference runs. A scale computed from a weight’s own group is computed once, offline, and stored. It costs no runtime work at all beyond being read alongside the weights it belongs to — and since a 4-bit weight-only kernel is already reading memory as fast as it can, an extra 0.156 bits per weight is a 0.156/4.156 ≈ 3.8% increase in the bytes it must move.

The accuracy argument is equally direct. A single scale for 45 million weights is set by the single largest weight among them, so one extreme value dictates the grid step for everything. Narrowing the set that shares a scale narrows the range that sets it, and a narrower range at the same number of levels is a finer step. Every quantization format in wide use is somewhere on this curve; the fine end is where the formats designed for 4-bit and below all live.

Why activations cannot

Activations get the coarse end of the same spectrum, and the reason is not memory — it is where the numbers have to be at the moment they are needed.

In a quantized matrix multiply, the accumulation runs over the reduction dimension. A scale that varies along that dimension cannot be factored out of the sum: you would have to rescale each partial product individually, inside the innermost loop, which is exactly the arithmetic the integer tensor cores exist to avoid. Scales that vary along the output dimensions — per output channel for weights, per token for activations — can be pulled out and applied once to the finished accumulator, which costs almost nothing.

So the practical rule is structural rather than empirical: weights may be quantized per-group along the reduction dimension because the dequantization happens as the weights are unpacked, before the multiply. Activations are usually quantized per-tensor or per-token because anything finer would land inside the accumulation. This is why activation quantization has to deal with outliers by moving them rather than by isolating them — see SmoothQuant and outlier features.

The KV cache is the third case and it sits between the two. Cached keys and values are activations, so their scales cannot be known offline, but unlike the activations inside a matmul they are stored rather than immediately consumed — which means a scale can be attached per head, per token or per block of the cache without ever landing inside an accumulation. This is why cache quantization schemes routinely use granularities that would be impossible for the activations feeding a linear layer, and why an 8-bit or 4-bit cache tends to cost less quality than the same bit width applied to activations in general.

Reading granularity off a config

Configurations rarely use the word “granularity”. What they use instead:

• group_size=128 — per-group weights. group_size=-1 in the GPTQ and AWQ configs means per-channel, one scale per output row.
• W8A8 with “dynamic per-token” activations — per-tensor or per-channel weights, per-token activations computed at runtime.
• Block-wise in bitsandbytes and in FP8 schemes — per-group under another name. NF4 uses blocks of 64; Transformers’ FineGrainedFP8Config documents a default weight_block_size of (128, 128), which is a two-dimensional block rather than a run along one axis.
• Super-blocks in llama.cpp’s K-quants — a hierarchy, where a block has its own quantized scale and a super-block has a scale over those. That structure is decoded in what K, M and S mean in a GGUF quant name.

When two formats claim the same bit width and produce different file sizes, granularity is almost always the reason, and the arithmetic in the second section above will account for the gap.