Loading a LoRA Adapter in llama.cpp on Top of a GGUF

llama.cpp does not merge a LoRA into the base weights. It loads the two low-rank matrices as separate tensors and adds their product into the forward pass, which is why a quantized base is fine and why you can change the strength without re-writing a 5 GB file.

What you need first

Three things, and the second is the one people skip. A base model in GGUF, quantized however you like. A PEFT adapter directory — the one containing adapter_config.json and adapter_model.safetensors — produced by training against that same base architecture. And the config files of the original Hugging Face base model, because the converter reads shapes and tokenizer metadata from them.

The third requirement surprises people who assumed the adapter was self-describing. It is not: adapter_config.json records a rank, an alpha, a list of target_modules and a base_model_name_or_path, but it does not record the hidden size or the layer count, so the converter has to go and read them. It does not need the base weights, only the configuration — llama.cpp’s own help text for the flag says as much.

Converting the PEFT adapter to GGUF

The script is convert_lora_to_gguf.py, in the root of the llama.cpp checkout. Its positional argument is the adapter directory; --base points at a directory of Hugging Face base model config files, and --base-model-id fetches that config from the Hub instead if you do not have it locally. If neither is given, the script falls back to the base_model_name_or_path recorded in the adapter config, and fails with Base model config is required. Please download the base model and add its path to --base when that lookup does not resolve.

# from the llama.cpp checkout
python convert_lora_to_gguf.py \
  --base ./Meta-Llama-3-8B-Instruct \
  --outtype f16 \
  --outfile ./adapters/support-tone-f16.gguf \
  ./peft-out/support-tone

# check the plan without writing anything
python convert_lora_to_gguf.py --base ./Meta-Llama-3-8B-Instruct --dry-run ./peft-out/support-tone

Keep --outtype f16. An adapter is small — tens of megabytes for a rank-16 adapter on an 8B model — so quantizing it buys almost nothing, and llama.cpp’s merge tool refuses quantized adapters outright with quantized LoRA adapters is not supported, please retry with f16 or f32.

Applying it at load time

Two flags. --lora FNAME applies the adapter at its recorded scale; --lora-scaled FNAME S applies it multiplied by S. Both imply --no-mmap, because the adapter has to be combined with weights the runtime is going to touch rather than faulted in lazily.

./llama-cli -m ./models/llama-3-8b-instruct-Q4_K_M.gguf \
  --lora ./adapters/support-tone-f16.gguf \
  -p "Customer says the invoice is wrong. Reply." -n 128 --temp 0

# half strength
./llama-cli -m ./models/llama-3-8b-instruct-Q4_K_M.gguf \
  --lora-scaled ./adapters/support-tone-f16.gguf 0.5 \
  -p "Customer says the invoice is wrong. Reply." -n 128 --temp 0

Note that the base here is Q4_K_M and that is not a compromise the tooling is tolerating. Since the adapter refactor that introduced GGUF adapters, llama.cpp computes Wx + B(Ax) during graph evaluation rather than folding BA into W beforehand, so the quantized base is never dequantized, modified and re-quantized. The old advice that a LoRA needs an f16 base belongs to the merge path, not this one. See the llama.cpp LoRA refactor pull request for the change that made this true.

The server takes the same flags, and adds one that is easy to trip over. llama-server accepts --lora and --lora-scaled at start-up and applies them to every request; it also accepts --lora-init-without-apply, which loads the adapters into memory at scale zero so that individual requests can turn them on. That second mode is what you want for serving several adapters over one resident base, and it is also a reliable way to spend an afternoon wondering why a correctly-loaded adapter has no effect — loaded and applied are separate states, deliberately.

Comparing against the unadapted base

1. Run the base alone at --temp 0 with a fixed --seed 1 and a prompt that sits squarely inside what the adapter was trained on. Save the output.
2. Run the identical command with --lora added and nothing else changed. Save that output.
3. Diff them. At temperature zero with a fixed seed, the base is near-deterministic, so any difference at all is the adapter. If the two files are byte-identical, the adapter did nothing to this prompt.
4. Run a third time with --lora-scaled ... 2.0. If output at double strength is also identical to the base, the adapter is not being applied — a scale that large should visibly distort generation.

That third run is the part worth keeping. An identical pair of outputs is ambiguous: it could mean the adapter is inert, or that this particular prompt was already answered the way the adapter would answer it. Cranking the scale removes the ambiguity, and a fuller version of that differential test is worth building once and reusing.

The four ways this fails

• Wrong base. LoRA tensor ‘...’ does not exist in base model (hint: maybe wrong base model?), or the shape variant of the same message. The adapter names tensors after the architecture it was trained on; a base with different projection dimensions cannot host it. A fine-tune of the instruct variant will usually load against the base variant and behave badly, which is worse than an error.
• Wrong file entirely. expect general.type to be ‘adapter’, but got: model means you passed a model GGUF to --lora. The converter writes a type marker into the adapter’s metadata precisely so that this is caught at load rather than producing garbage.
• A converted adapter with no base config. The conversion step, not the load step: Base model config is required. Please download the base model and add its path to --base. The adapter names a base in base_model_name_or_path that the converter could not resolve locally or on the Hub — usually a private path from whoever trained it. Pass --base or --base-model-id yourself.
• An adapter from the removed trainer. lora_a tensor is not transposed (hint: adapter from "finetune" example is no longer supported). llama.cpp’s old in-tree finetune example produced a layout the current loader will not read. Retrain with PEFT, or keep an old build; there is no converter for it.

If you want the adapter permanently baked in instead — for distribution, or for a runtime that cannot take adapters — llama-export-lora writes a merged GGUF, and the cost of not merging is small enough that most people should not bother.