Group Size in GPTQ and AWQ Quantization

Group size is the parameter people change first and understand last. It is not a quality dial with a mysterious cost — the cost is exactly computable, it is measured in bits per weight, and once you can compute it the choice between 32, 64 and 128 stops being folklore.

What a group is

A 4-bit weight is an integer from 0 to 15. To turn it back into a real number you need two more values: a scale, which says how much one integer step is worth, and a zero point, which says which integer means zero. Those two are shared by a set of weights, and that set is the group.

With group_size=128, consecutive runs of 128 weights along the input dimension of a row share one scale and one zero point. With group_size=-1 in both GPTQConfig and AwqConfig, the group is the whole row — one scale per output channel, the coarsest option those tools offer. The direction the grouping runs in matters: it is along the reduction dimension, so all the weights in a group multiply different input channels of the same output. That is precisely the dimension along which activation magnitudes vary wildly, which is why the size of the group has any effect at all.

The storage cost, worked

Assume the common packing: a 16-bit scale and a 4-bit zero point per group, with 4-bit weights. Overhead per weight is the per-group metadata divided by the group size.

overhead(g) = (16 + 4) / g  bits per weight

g = 128 →  20/128 = 0.156  →  4.156 bits per weight
g =  64 →  20/64  = 0.313  →  4.313 bits per weight
g =  32 →  20/32  = 0.625  →  4.625 bits per weight
g =  -1 →  ~0                →  4.000 bits per weight

Turn that into gigabytes for a 7B model, counting only the quantizable linear weights:

7.0e9 * 4.156 / 8 = 3.64 GB   (g=128)
7.0e9 * 4.313 / 8 = 3.77 GB   (g=64)
7.0e9 * 4.625 / 8 = 4.05 GB   (g=32)

So the whole span from 128 down to 32 is about 0.41 GB on a 7B model — around 11%. On a 70B model the same 0.469 bits per weight is about 4.1 GB, which is the difference between fitting on a 48 GB card and not. The arithmetic scales linearly with parameter count, so the decision gets more expensive exactly as models get large enough for it to be tight.

What you buy with a smaller group

The scale of a group is set by the range of the weights in it, and the grid step is that range divided by 15. A group whose weights span [-0.02, 0.02] gets a step of about 0.0027; the same group merged with a neighbouring one containing a single weight of magnitude 0.5 gets a step of about 0.067, twenty-five times coarser. Every weight in the merged group now rounds twenty-five times as badly, for the sake of one outlier.

Smaller groups isolate that damage. The outlier lands in a group of its own 32 or 64 neighbours instead of 128, and the other groups keep their tight ranges. That is the entire accuracy mechanism, and it explains the two things people notice about group size:

• The benefit is larger at lower bit widths. At 4 bits you have 16 levels to lose; at 3 bits you have 8 and at 2 bits you have 4. A coarse scale is proportionally more destructive when there are fewer levels to absorb it, which is why 3-bit and 2-bit bakes almost always use smaller groups than 4-bit ones.
• The benefit depends on how outlier-heavy the model is. A model whose weights are cleanly distributed loses little to a large group. One with pronounced outlier structure loses a great deal. This is why the same group size behaves differently across model families and why a rule copied from one model is not evidence about another.

Why 128 became the default

Both GPTQConfig and AwqConfig in Hugging Face Transformers default to group_size=128 and both describe 128 as the recommended value. The reasons are not purely about error.

• 0.156 bits per weight is cheap. Under 4% overhead on a 4-bit weight is small enough that nobody argues about it, while still being 128 times finer than one scale per row.
• Kernels are written for it. Mixed-precision matmul kernels load weights in tiles, and a group boundary that lines up with a tile boundary means the scale can be loaded once per tile and applied in registers. A group size that does not divide the tile evenly forces either a reload or a slower path. 128 divides comfortably into the shapes these kernels use, which is why 4-bit group-128 is the configuration with the best kernel coverage across runtimes.
• It is what the checkpoints are baked at. Most published GPTQ and AWQ files on Hugging Face use 128, so it is the configuration with the most inference paths that have been exercised by other people.

When to deviate

Three situations genuinely justify moving off 128, and one common one does not.

• Going below 4 bits. If you are baking at 3 bits or less, a smaller group is doing real work, and the extra 0.16 to 0.47 bits per weight is a small fraction of what you saved by dropping the bit width. Alternatively, spend the budget non-uniformly: mixed-precision quantization allocates it per layer rather than per group.
• A model with known outlier structure. Where a model’s weights have concentrated extremes, a smaller group contains the damage rather than spreading it.
• You are memory-rich and quality-bound. If the model already fits with room to spare, group 32 costs you a fraction of a gigabyte for a strictly finer approximation. Check first that your kernel supports it at speed.

The situation that does not justify it is “the output looked wrong so I halved the group size”. Group size changes the rounding of every weight slightly; it does not fix a broken chat template, a wrong stop token, or sampling parameters set badly. Rule those out before spending a bake on it.