GPTQ Explained: How Post-Training Calibration Works

GPTQ is normally introduced as “one-shot post-training quantization using approximate second-order information”. That sentence is accurate and tells you nothing about what happens to your weights. What happens is a loop over the columns of one weight matrix, and it is short enough to follow all the way through.

The objective is one layer, not the model

GPTQ never evaluates the model’s loss. It takes one linear layer at a time and asks a purely local question: given the inputs X that this layer actually receives, find quantized weights Ŵ that minimise the squared error in the layer’s output, not in its weights. Formally it minimises ||WX − ŴX||², where X is a matrix of activations collected by running calibration text through the model.

That framing is why the method needs so little data and no gradients. You are not retraining anything; you are solving a least-squares reconstruction problem for one matrix, with the grid of representable values as the constraint. The GPTQ paper (Frantar, Ashkboos, Hoefler and Alistarh, ICLR 2023) uses 128 randomly chosen 2048-token segments of C4 for exactly this, and nothing about the objective wants more. What the calibration set has to do is described in what a calibration dataset actually does.

Why round-to-nearest is the wrong answer

The naive method, round-to-nearest, snaps every weight independently to the closest representable value. It minimises weight error perfectly and output error badly, because the two are not the same quantity. A weight that multiplies an input channel with a typical magnitude of 40 contributes twenty times more output error per unit of rounding than a weight multiplying a channel with a magnitude of 2. Round-to-nearest cannot know this, because it never looks at X.

GPTQ inherits its answer from Optimal Brain Quantization, itself descended from the Optimal Brain Surgeon pruning literature: after you round one weight and incur an error, you are still free to adjust every weight you have not rounded yet. Nudge them so that the layer’s output moves back towards where it was. The right nudge is given by the inverse of the Hessian of the reconstruction objective, which for this objective is simply H = 2XXᵀ + λI — an outer product of the calibration activations, plus a dampening term on the diagonal.

The loop, column by column

For one weight matrix, with columns indexed by input channel:

for j in columns(W):
    # 1. round column j to the grid (scale/zero from its group)
    Q[:, j] = quantize(W[:, j])

    # 2. the error this rounding introduced, normalised by the
    #    Hessian's own confidence in column j
    E[:, j] = (W[:, j] - dequantize(Q[:, j])) / Hinv[j, j]

    # 3. push that error into every column not yet quantized
    W[:, j+1:] -= E[:, j] * Hinv[j, j+1:]

Three things are worth reading twice. The division by Hinv[j, j] in step 2 is what makes the correction proportional to how much this column matters — a column the calibration data barely exercises gets a small correction, because moving it would not help. Step 3 only ever writes to columns to the right, which is why the loop can run once and never revisit. And the weights being written in step 3 are full-precision values that have not been rounded yet, so the compensation is genuinely free: it costs no extra bits in the output file.

After the last column, every weight has been rounded and the accumulated error has been spread over the columns that could absorb it. The stored artefact is the packed integer codes plus, per group, a scale and a zero point.

The three shortcuts that make it fast

Optimal Brain Quantization as published re-derives the Hessian inverse after every single weight, which is why it was demonstrated on models of a few hundred million parameters and no further. GPTQ makes three changes, and together they are the reason the method finishes at all.

• A fixed column order. OBQ picks the next weight greedily, which means a different order for every row of the matrix and therefore a different Hessian inverse for every row. GPTQ quantizes all rows in the same column order, so one H⁻¹ is computed once and shared by the entire matrix. The paper’s observation is that the greedy order is worth very little on large layers, and this is the change that removes a factor of the row count from the cost.
• Lazy batch updates. Step 3 above touches the whole remaining matrix after every column, which is memory-bandwidth-bound and leaves the GPU idle. GPTQ instead processes a block of columns (128 in the reference implementation), accumulating their errors against the block only, then applies one bulk update to everything to the right of the block. Same result, one large matrix operation instead of 128 small ones.
• A Cholesky reformulation. Repeatedly updating an inverse Hessian in float accumulates error, and on large layers it accumulates until the matrix stops being positive definite and the whole thing produces garbage. Since the column order is now fixed, every inverse the loop will ever need is known in advance, and they can all be read off a single Cholesky decomposition computed once in a numerically stable routine.

The dampening term is the fourth piece of numerical insurance: damp_percent adds a fraction of the mean Hessian diagonal to the diagonal before inversion, so a layer whose calibration activations happen to be rank-deficient still inverts. It defaults to 0.1 in Hugging Face’s GPTQConfig, and a run that fails with a Cholesky or singular-matrix error is usually asking for a larger value.

The knobs you actually set

Everything above is fixed by the algorithm. These are the parameters a quantization run exposes, with the defaults Hugging Face Transformers’ GPTQConfig documents:

from transformers import GPTQConfig

cfg = GPTQConfig(
    bits=4,              # 2, 3, 4 or 8
    group_size=128,      # -1 for one scale per output row
    desc_act=False,      # "act-order": quantize columns in
                         # descending Hessian-diagonal order
    damp_percent=0.1,    # diagonal dampening before inversion
    sym=True,            # symmetric grid, zero point fixed at 0
    true_sequential=True,
    dataset="c4",
)

• group_size decides how many weights share one scale. It is the single biggest lever on the size/quality trade and has its own arithmetic.
• desc_act is the one shortcut above put back. Quantizing columns in descending order of their Hessian diagonal handles the most-used channels while the most correction budget is still available, which helps quality; the cost is that the column permutation must be undone at inference, which historically made some kernels slower. Hugging Face’s documentation describes it as significantly faster at inference when off, with perplexity slightly worse.
• bits below 4 is where GPTQ separates most sharply from round-to-nearest, and also where the format starts needing help — see mixed-precision quantization.

The output of all this is a file of packed 4-bit integers that no matrix-multiply unit can consume directly. Making it fast again is a separate problem, solved in the kernel — see the Marlin kernel. The other major post-training method starts from a completely different premise about which weights matter: AWQ.