Testing Whether a LoRA Adapter Is Actually Doing Anything

llama_adapter_lora_init_impl: loaded 224 tensors from lora file means the file parsed and its tensors matched the base. It does not mean the adapter is affecting your output. This page is about the gap between those two facts, because that gap prints nothing.

The log line that is not proof

Most pages about a broken adapter start from an exception. This one cannot, and that is deliberate: the failure people actually search for is the one where every command succeeded. llama.cpp prints loading lora adapter from ‘...’, then a line per adapted tensor of the form lora for ‘blk.0.attn_q.weight’ -> ..., then a buffer size, then the tensor count. Ollama prints nothing at all. Neither says whether the result reached the sampler.

There is exactly one number in that output worth reading, and it is the tensor count. Compare it against what you expect from the adapter config: an adapter with target_modules of q_proj and v_proj on a 32-layer model has two adapted projections per layer and two matrices each, so 128 tensors. A count far below that means the loader matched fewer modules than the adapter contains. A count of zero means it matched none, and everything after it is the base model answering.

The reason this needs spelling out is that the two states look identical from outside. A model answering well because an adapter is shaping it and a model answering well because the base was already competent on your prompts produce the same kind of output, and the second is what you get when the adapter drops out of the pipeline. There is no warning for “the thing you configured is not affecting the result”, because from the runtime’s point of view nothing went wrong: the file parsed, the tensors matched, the scale was a valid number. Every component reported success and the composition of them did nothing.

Four ways it loads and does nothing

• The scale is zero. --lora-scaled adapter.gguf 0 is a valid command that applies nothing. So is a per-request scale of 0.0 in llama-server, which is also the default state for adapters loaded under --lora-init-without-apply.
• The adapter was never activated per request. llama-server lets a request carry its own adapter selection. A client that does not send that field gets whatever the server-wide default is, which may be nothing.
• The adapter is real but weak on this prompt. An adapter trained to change tone on support replies will not change a factual answer about arithmetic. This is not a bug and it is the most common cause of “it isn’t working”.
• The training barely moved anything. A run that stopped early, used a tiny learning rate, or targeted modules the task does not route through produces an adapter whose matrices are near zero. It loads perfectly and changes nothing, anywhere.

The differential test

The test has to hold everything constant except the adapter, and has to include a run that would be obviously wrong if the adapter were applied. That second part is what separates this from the usual advice.

1. Pick five prompts from the training distribution — the kind of input the adapter was tuned on, not a generic question. Write them to a file, one per line.
2. Run the base with greedy decoding and a fixed seed: --temp 0 --seed 1 -n 128. Save each output.
3. Run again with the adapter at its intended scale, changing nothing else. Save.
4. Run a third time at a deliberately absurd scale — --lora-scaled adapter.gguf 4.0. Save.
5. Diff all three sets.

#!/usr/bin/env bash
set -euo pipefail
BASE=./models/llama-3-8b-instruct-Q4_K_M.gguf
LORA=./adapters/support-tone-f16.gguf
run () { ./llama-cli -m "$BASE" "${@:2}" -f prompts.txt \
           --temp 0 --seed 1 -n 128 --no-display-prompt > "$1"; }

run out.base
run out.lora   --lora "$LORA"
run out.loud   --lora-scaled "$LORA" 4.0

diff -q out.base out.lora  && echo "ADAPTER HAD NO EFFECT AT SCALE 1"
diff -q out.base out.loud  && echo "ADAPTER NOT APPLIED AT ALL"

Reading the result

Three outcomes, and each points somewhere different.

• Base and 4.0 identical. The adapter is not being applied. A scale of four on a real adapter degrades output visibly, often into repetition or broken syntax. If nothing changed, the update is not reaching the forward pass — check the scale, the per-request selection, and that the tensor count in the load log was not zero.
• 4.0 differs, scale 1.0 identical to base. The adapter is applied and its effect is genuinely small on these prompts. Either the training barely moved the weights or your prompts are outside what it was tuned for. Try prompts closer to the training data before concluding the adapter is broken.
• All three differ. Working as intended. Now the question is whether the difference is an improvement, which is an evaluation problem rather than a loading problem, and needs a rubric and real examples rather than a diff.

One caution on the greedy assumption: --temp 0 with a fixed seed makes generation near-deterministic, not bit-deterministic. Different thread counts, batch sizes or backends can reorder floating-point accumulation and change a token near a tie. Keep every other flag identical between runs, and if you see a one-token difference in an otherwise identical output, suspect that before suspecting the adapter — the sampler settings have to be pinned for the comparison to mean anything.

Keeping it as a check

The value of this test is not the one time you run it after a conversion. It is having it as a fixture, because the pairing between an adapter and a base is enforced by nothing: a base tag that gets repointed, a Modelfile whose relative path stops resolving, a server restarted without its adapter flag, all produce a service that answers normally and has quietly reverted to the base model.

Store out.base and out.lora as golden files and assert two things in CI: that the adapted output still differs from the base, and that it still matches the stored adapted output. The first catches the adapter falling out of the pipeline; the second catches the base changing underneath it. Both failures are otherwise invisible until a user notices the answers got blander, which is not a bug report anyone files.