Running Out of Memory During Quantization, Not Inference

The finished 4-bit model will run comfortably on this machine. The process that produces it will not, and that is not a contradiction — quantization is a different workload with a different peak.

What the failure looks like

Two shapes, depending on which memory ran out. On the GPU:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.06 GiB.
GPU 0 has a total capacity of 79.15 GiB of which 1.28 GiB is free.

On the host, there is often no exception at all — the Linux out-of-memory killer terminates the process and the shell prints a single word:

Killed

The second is the more confusing one, because there is no traceback and nothing in the application log. Confirm it rather than guessing: dmesg -T | tail or journalctl -k | tail will show the kernel recording which process it chose and how much it was using. On macOS the equivalent appears in the system log and the shell reports zsh: killed, which is the same signal from a different supervisor — see the killed-process page.

Note which stage it died in. Quantization runs in phases — load, then calibrate, then quantize layer by layer, then save — and the phase names the cause.

Why quantizing costs more than running

Inference reads a quantized weight and uses it. Quantization has to decide what the quantized weight should be, and to do that it needs the original alongside the machinery for choosing. Concretely, three things are resident that never coexist at inference time:

• The source weights at full precision. You are quantizing from 16-bit, so the input is roughly four times the size of the output. A 70B model at bf16 is about 70e9 × 2 = 140e9 bytes, or ~130 GiB, before anything else happens.
• Calibration activations. Data-driven methods run sample text through the model and keep the activations for the layer being processed. That is a batch of real tensors, sized by sequence length times batch size times hidden dimension.
• The method’s own statistics. For GPTQ this is the term that dominates, and it is worth its own section.

Add the output being assembled, and the peak lands well above both the input model and the output model.

The Hessian, which is the expensive part

GPTQ chooses each layer’s quantized weights by minimising the error in that layer’s output, which requires a matrix of second-order statistics over the layer’s inputs. For a linear layer with input dimension d, that matrix is d × d, accumulated and inverted in float32. The arithmetic for a large model’s wide layers:

d = 8192   ->  8192 x 8192 x 4 bytes = 268,435,456 bytes ~= 256 MiB
d = 16384  -> 16384 x 16384 x 4 bytes = 1,073,741,824 bytes = 1 GiB

That is per layer, and the Cholesky inversion needs working space alongside it. This is why reported failures cluster on the inversion step rather than on loading, and why they get worse for models with wide feed-forward dimensions rather than simply for models with more parameters. It is also why the cost is quadratic in width: doubling the hidden dimension quadruples the statistic.

AWQ works differently — it searches per-channel scales using activation magnitudes rather than building an inverse Hessian — so its peak is driven by the calibration batch rather than by a d × d matrix. If GPTQ will not fit and you have calibration data, activation-aware methods are often the cheaper pass on the same hardware. Their own failure modes are separate; the AWQ batch-size assertion is one.

Bitsandbytes-style round-to-nearest quantization has no calibration phase at all and therefore no such peak, which is exactly why it is the method that works on hardware where the others do not — at some cost in quality, discussed in choosing a quantization level.

The GGUF path, which fails differently

llama.cpp’s llama-quantize does not use a GPU and does not need calibration, so it never fails the way GPTQ does. It fails on host RAM, and the project’s own documentation is explicit about why: the models are fully loaded into memory, so you need enough RAM to hold them as well as disk space to save them.

That means the binding constraint is the input file. Converting a 70B checkpoint to GGUF at 16-bit produces roughly a 130 GiB file, and quantizing that file wants comparable RAM. The conversion step before it — convert_hf_to_gguf.py — is a second, separate peak on the same machine.

If you also want an importance matrix for better low-bit quality, that is a third pass: llama-imatrix runs the unquantized model over calibration text, which needs the same memory as inference on the unquantized model. Doing all three on one machine is what most out-of-memory reports on this path actually are.

Fitting the job on the machine you have

1. Establish which memory ran out. A CUDA exception means VRAM; Killed with a kernel log entry means host RAM. Every remaining step differs between the two.
2. Add swap before adding hardware, for the host case. A quantization pass is throughput-bound rather than latency-bound, so spilling to a fast NVMe disk is slow but finishes, and a job that finishes overnight beats one that dies in ten minutes. This is poor advice for inference and good advice here.
3. Shrink the calibration batch and sequence length before anything else on the GPU side. Both are direct multipliers on the activation term and neither changes the resulting weights much.
4. Use the library’s offloading so only the layer being quantized is resident on the device. Layer-by-layer processing is what makes quantizing a model larger than your GPU possible at all; if the tool offers a CPU-offload device map, that is the switch.
5. Or do not quantize it yourself. The overwhelmingly common case is a model somebody has already quantized and published, and the memory required to download a finished GGUF or AWQ checkpoint is zero. Reserve doing it yourself for a fine-tune or a model nobody has covered.