A GGUF Model Loads but Outputs Garbage

The file loaded without a complaint, which means it is structurally valid and its tensors are the right shapes. Whatever is wrong is a setting, and the shape of the nonsense narrows it to one of four.

Two kinds of garbage

Before changing anything, classify the output. The two look similar in a chat window and have almost disjoint causes.

• Token soup. Repeated single characters (“GGGGGGGG”), random Unicode, fragments of unrelated languages, output that was never English. This is the model computing wrong numbers — a numerical or tokenizer problem, and usually broken from the very first token.
• Fluent but wrong. Grammatical text that ignores the instruction, answers a question you did not ask, continues your prompt instead of replying, talks to itself, or emits literal marker strings like <|im_start|> and <|eot_id|>. This is the model computing correctly on input that was assembled wrongly — a formatting problem.

A third possibility that is neither: coherent output that degrades into repetition after a while. That is a sampling and context issue rather than a load-time misconfiguration, treated in repetition loops.

Cause one: the chat template

Instruction-tuned models are trained on one exact string format, with specific special tokens marking role boundaries. Send them a different format and you are prompting a model outside its training distribution; it will produce fluent, confident, off-target text. This is the single most common cause of the second kind of garbage.

The tells are specific and easy to spot:

• Special tokens appearing as visible text in the output, which means they were tokenized as ordinary characters rather than as the single control tokens they are.
• The model continuing your message rather than replying, which means no assistant turn was opened.
• The model producing several turns of a conversation with itself, which means the stop token for a turn was never applied — the flip side of an empty response, where the stop fires too early.

GGUF files carry the template in their metadata, and a runtime that applies it will get this right. Two ways it goes wrong: the file was converted without the template, or your client is applying its own. Dump the metadata and look:

pip install gguf
gguf-dump --no-tensors model.gguf | grep -i -E "chat_template|bos|eos|pre"

If the template is absent, supply the correct one from the model card — llama.cpp accepts a named built-in or a Jinja template — rather than hoping a generic default matches. How chat templates work covers the general mechanism, and the Llama 3 template tokens is a worked example of what these strings look like.

Cause two: rope settings

Rotary position embeddings encode token positions by rotating query and key vectors at frequencies derived from a base value. Get the base or the scaling wrong and every position in the sequence is misrepresented — the model is being told a different sentence order than the one you wrote.

The characteristic signature is position-dependent: coherent for a short prompt, degrading as the context grows, and collapsing entirely past some length. If output is fine at 500 tokens and garbage at 5,000, look here first and nowhere else. It has two origins:

• Missing or wrong metadata in the file. Long-context models that use a non-default base, or a scaling scheme applied at fine-tune time, need those values recorded at conversion. A conversion that predates support for the scheme writes defaults, and the file then loads happily and behaves badly at length. Models with extended windows are the usual victims — see Llama 3.1’s rope scaling for a concrete scheme.
• An override you set. Passing --rope-freq-base or a scaling factor copied from a thread about a different model overrides correct metadata with wrong values. Remove every rope flag and retest before adding any back.

Read what the file declares, then compare against the model card:

gguf-dump --no-tensors model.gguf | grep -i rope

A related failure with a similar look: exceeding the context the model was trained for, which is the local context-length error when the runtime notices and degraded output when it does not.

Cause three: the tokenizer

If the text is broken from the very first token, suspect tokenization. A GGUF embeds its own vocabulary and pre-tokenizer configuration, and a mismatch between what the file declares and what the model was trained with corrupts the input before any weights are consulted.

There is a well-known historical instance of this. When llama.cpp added support for byte-pair pre-tokenizers that differed from its default, it began warning loudly on files converted before that support existed:

llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************

If a warning of that kind appears in your load log, believe it: the fix is a file reconverted with a current build, not a settings change. More generally, any conversion done with tooling older than the model is suspect, and re-downloading from a publisher who reconverted after the relevant fix is cheaper than debugging.

Cause four: the backend or the quant

The remaining case is that the arithmetic itself is wrong on this hardware path. Reports of this cluster on specific backend and quantization combinations — a particular quant type miscomputing under one GPU backend while being fine on CPU and fine under another backend — and they are genuine bugs, not misconfiguration.

The test is decisive and takes one command. Run the same file, same prompt, same seed, with no GPU offload at all:

llama-cli -m model.gguf -ngl 0 -p "Write one sentence about rain." --seed 1

Coherent on CPU and garbage on GPU means the backend or the kernel for that quantization type is at fault. Update llama.cpp; if that does not help, try a different quantization type or a different backend build, and check the project’s issues for your combination. Garbage in both places clears the backend entirely and sends you back to the three causes above.

Extremely low-bit quantizations deserve a mention here: at two or three bits, quality loss stops being subtle and can look like a bug. That is the mechanism working as designed, and choosing a quantization level covers where the cliff sits.

Triage order

1. Classify the garbage. Token soup goes to the tokenizer and backend checks; fluent-but-wrong goes to the template check.
2. Read the load log from the top. Warnings about pre-tokenizers, missing metadata or rope defaults appear there and answer the question for free.
3. Remove every sampling and rope flag you set. Test with defaults and a fixed seed. Adding settings back one at a time is faster than reasoning about them together, and the sampling parameters have their own effects worth isolating.
4. Run with -ngl 0. This partitions the problem into hardware and not-hardware in one step.
5. Dump the metadata and compare the template, the special tokens and the rope values against the model card. Where they disagree, the file is wrong and reconversion or a different upload is the fix.