LoRA Adapter Compatibility Across Quantization Formats

“Will this adapter work with my quantized model?” is two questions wearing one coat. One is about file formats and has a crisp answer; the other is about numerical quality and has a nuanced one. They get conflated constantly and the conflation is why the advice online contradicts itself.

Two questions people ask as one

The first question is whether the loader can read the adapter file at all. That is decided by the container and the tensor naming convention, and it has nothing to do with how the base is quantized. An adapter is a set of small dense matrices; it is stored in a container, and a loader reads one container or another.

The second question is whether adding a full-precision low-rank update to a base whose weights have been rounded still produces the behaviour the adapter was trained to produce. That is a quality question, it has degrees, and the answer depends on how aggressively the base was quantized and where the adapter’s effect lives. Keep the two apart and the picture stops being confusing.

It also helps to know what you are holding before you ask whether it fits. A PEFT adapter is a directory: adapter_config.json, whose peft_type is LORA and which records r, lora_alpha, target_modules and base_model_name_or_path, alongside adapter_model.safetensors. A GGUF adapter is a single file whose metadata carries an adapter type marker and the same rank and alpha as key-value pairs. If you were handed “a LoRA” and cannot tell which you have, the presence or absence of adapter_config.json settles it in one command.

What each loader accepts

• llama.cpp — a GGUF adapter, and only that. The file must carry general.type = adapter and adapter.type = lora in its metadata, which is what convert_lora_to_gguf.py writes. A PEFT adapter_model.safetensors handed to --lora is not read.
• Ollama — a GGUF adapter file, or a Safetensors adapter directory for the architectures its Modelfile reference lists as supported. The Safetensors path is a convenience that converts on your behalf at ollama create time; the GGUF path always works.
• Transformers with PEFT — the PEFT directory form, adapter_config.json plus adapter_model.safetensors. It does not read GGUF adapters, because it does not read GGUF weights as a native format either.
• vLLM and similar servers — PEFT directories, usually with a maximum rank and a list of adaptable modules set at server start. This is the only common path that serves many adapters over one resident base.

Why a GGUF adapter cannot meet a GPTQ base

Because nothing loads both. It is not that the two are numerically incompatible — it is that a GGUF adapter is a llama.cpp artefact and llama.cpp does not load GPTQ weights, while the Python-side runtimes that do load GPTQ do not read GGUF adapters. The pairing has no host.

The formats themselves are also nothing alike underneath, which is why nobody has bridged them casually. A GPTQ checkpoint stores qweight as low-bit values packed into int32 words, alongside scales, qzeros and a g_idx permutation, grouped along the input dimension — typically 128 weights per group. A GGUF k-quant such as Q4_K stores super-blocks of 256 weights divided into eight sub-blocks of 32, with the per-sub-block scales and minima themselves quantized to 6 bits, which llama.cpp’s quantize documentation gives as 4.5 bits per weight overall. Different block sizes, different axis, different metadata. The full argument is on why GPTQ weights do not convert to GGUF.

The practical route, if you have a PEFT adapter and want it on a quantized base you do not yet have: convert the adapter once with convert_lora_to_gguf.py, quantize the original fp16 base to GGUF separately, and pair those. You are not converting between quantization schemes at any point, which is exactly why it works.

A quantized base is usually fine

On the quality question, the mechanism matters. When a runtime applies an adapter at inference time it computes Wx + B(Ax): the quantized base is read as-is and the adapter contributes in higher precision. Nothing about the base is dequantized, altered and re-rounded, so the adapter’s update is not itself quantized away. The older warning that a LoRA needs an f16 base is about the merge path, where BA is folded into W and the result is quantized again — there, a small update to a weight that is already rounded to one of sixteen levels can round straight back to where it started.

Where the residual risk sits is at the aggressive end. A base at 2 or 3 bits has lost enough of the structure the adapter was trained against that the pairing can behave unpredictably, and quantization error is not evenly distributed — it concentrates on outlier features that matter disproportionately, which is also why QLoRA trains the adapter against the quantized base rather than the original. If the adapter matters and the base is heavily quantized, test the pairing rather than assuming it.

Reading the error you get

llama.cpp’s adapter loader is specific, which makes diagnosis quick if you know what each message means.

• expect general.type to be ‘adapter’, but got: model — wrong file. You passed a model GGUF to --lora.
• model arch and LoRA arch mismatch — right kind of file, wrong family. The adapter was converted against a different architecture than the base you loaded.
• LoRA tensor ‘...’ does not exist in base model (hint: maybe wrong base model?) — same family, wrong size or variant. Check that the base is the exact model named in the adapter’s base_model_name_or_path.
• tensor ‘...’ has incorrect shape (hint: maybe wrong base model?) — usually a vocabulary change. A fine-tune that added tokens resized the embedding, and the adapter now expects a base that has them.
• No error and no observable change — the failure mode with no message. Test for it explicitly.