Why GPTQ Weights Usually Cannot Be Converted Directly to GGUF

ValueError: Can not map tensor ‘model.layers.0.self_attn.q_proj.qweight’ is not a missing feature in the converter. It is the converter telling you that the file contains packed integers and a permutation index where it expected a floating-point weight matrix.

The error

You have a GPTQ repository — safetensors files, a quantize_config.json, maybe -GPTQ in the name — and you run llama.cpp’s converter over it:

$ python convert_hf_to_gguf.py ./Model-7B-GPTQ --outfile model.gguf --outtype f16
INFO:hf-to-gguf:Loading model: Model-7B-GPTQ
INFO:hf-to-gguf:gguf: loading model part 'model.safetensors'
Traceback (most recent call last):
  ...
ValueError: Can not map tensor 'model.layers.0.self_attn.q_proj.qweight'

The same failure surfaces on .qzeros, .scales or .g_idx depending on iteration order, and the identical message shape appears for other quantized formats — FP8 checkpoints fail on weight_scale_inv, for instance. In every case the cause is the same: the checkpoint does not contain the tensors the converter knows how to name.

You can see it coming before you run anything. A GPTQ repository has a quantize_config.json, or a quantization_config block in config.json naming gptq as the method with a bits and a group_size; its safetensors index lists qweight rather than weight for every linear layer, and it is roughly a quarter the size a 16-bit checkpoint of that parameter count would be. Any one of those is enough to know the conversion will not run.

What the converter is looking for

convert_hf_to_gguf.py walks the state dict and maps each Hugging Face tensor name to a GGUF name — model.layers.0.self_attn.q_proj.weight becomes blk.0.attn_q.weight and so on. The mapping table is per architecture and it is a table of weights. When it meets q_proj.qweight there is no entry, because qweight is not a weight in the sense the table means: it is a container for packed quantized values that only becomes a weight in the company of three other tensors.

The converter is also not a quantizer. Its job is to read floating-point tensors and write them into a GGUF at f32, f16, bf16 or Q8_0; the real quantization happens afterwards in llama-quantize, which reads a high-precision GGUF and produces a k-quant one. There is no point in the pipeline that takes already-quantized integers as input, and adding one would not be a parser change.

Two 4-bit layouts with nothing in common

Both formats are called 4-bit and the resemblance ends there.

• GPTQ, from Frantar et al. (2022), stores four tensors per linear layer. qweight holds low-bit values bit-packed into 32-bit words; scales and qzeros hold a scale and a zero point per group along the input dimension, with a group size of 128 by default; g_idx records which group each input column belongs to, because GPTQ may reorder columns by activation importance. Reconstruction is per group, on the input axis, with a zero point.
• GGUF Q4_K stores one tensor per layer in a block format. Per llama.cpp’s quantize documentation, it uses super-blocks of 256 weights containing eight blocks of 32, with each block’s scale and minimum themselves quantized to 6 bits, giving 4.5 bits per weight overall. Blocks run along the flattened tensor, not along a chosen axis, and there is no permutation index.

So a transcoder would have to undo a 128-wide grouped affine quantization with an arbitrary column permutation, then re-derive 32-wide block scales and minima, then quantize those scales to 6 bits. Every one of those steps is a re-quantization. There is no bit-level correspondence to exploit, which is why nobody has written the direct path even though both formats are open and well documented.

Why dequantizing first is a bad trade

You can dequantize a GPTQ checkpoint back to fp16 — the arithmetic is defined and several libraries expose it — and then convert that to GGUF and quantize it. It runs. It is still usually the wrong thing to do.

The reason is that quantization error does not cancel. GPTQ chose its 4-bit grid to minimise error against the original weights, using calibration data. Dequantizing gives you fp16 numbers that sit exactly on that grid — they look like full precision and carry none of the information that was discarded. Quantizing those to Q4_K then picks a second grid, and the errors of the two schemes compound rather than cancel, because the second quantizer is optimising against an already damaged target and has no way to know it.

You also lose what k-quants are good at. llama-quantize assigns different bit widths to different tensors, and can use an importance matrix from calibration text to decide where precision matters. Fed a dequantized GPTQ model, all of that machinery is optimising against rounded values. The output file will be the right size and it will be worse than either honest 4-bit model.

The route that works

1. Look for an existing GGUF of the same model first. Popular models are converted from the original weights by people who had them, and that conversion is strictly better than anything you can derive from a GPTQ file.
2. Otherwise get the original fp16 or bf16 weights — the unquantized repository the GPTQ build was made from. It is named in the GPTQ repository’s own model card. If it is gated, request access; the licence is the mechanism by which these weights are available at all.
3. Convert the container: python convert_hf_to_gguf.py ./Model-7B --outfile model-f16.gguf --outtype f16. This step is lossless in the sense that matters — it changes the file layout, not the numbers.
4. Quantize once, from that: ./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M. One quantization step, applied to unrounded weights.
5. If you have a GPTQ file you cannot replace — a fine-tune whose fp16 weights were never published — run it in a runtime that reads GPTQ natively rather than converting it. That constraint is also why a GGUF adapter cannot meet a GPTQ base.

The general rule this is a case of: convert containers freely, convert quantization schemes never. Any path that goes from one lossy encoding to another lossy encoding is paying twice for one benefit, and the honest fix is always to go back to the highest-precision weights you can legally obtain.