A GGUF Model Returns an Empty Response Every Time

The model loads, the timings print, the token count is zero and the content is an empty string. No error anywhere. That is a model that predicted a stop token first, which is nearly always a prompt-format problem rather than a broken file.

What zero tokens means

Generation stops for one of three reasons: the token budget ran out, a stop string matched, or the model emitted an end-of-generation token. An empty response with a normal load and normal timings means the third happened on step one. The sampler picked a stop token as the most likely continuation of your prompt, and the loop exited before producing anything visible.

That is not a malfunction. It is the model doing exactly what it was trained to do, given a prompt that — as far as it can tell — is already a completed turn. An instruction-tuned model has seen millions of examples where a particular byte sequence marks the end of an assistant message. Put that sequence, or something the model reads as equivalent, in the wrong place and the highest-probability next token is “stop”.

Distinguish this from output that arrives and is nonsense, which is a different fault with different causes and lives in the garbage-output page. Empty is a stop-token problem; garbled is usually a quantisation, tokeniser or sampling problem.

Cause 1: the wrong chat template

Every instruction-tuned model has a template that turns a list of messages into one token sequence. Llama 3 uses header tokens and <|eot_id|> between turns; ChatML models use <|im_start|> and <|im_end|>; Mistral uses bracketed instruction markers; Gemma uses its own turn markers. The template is not decoration — it is the format the model was trained to continue.

Three ways this goes wrong and produces an empty answer:

• Applying two templates. A front end formats the messages, then the server applies the GGUF’s embedded template on top. The result contains a complete assistant turn followed by an end-of-turn marker, so the model correctly decides there is nothing left to say. This is the most common instance and it appears when a client that was building its own prompt is pointed at a server that also templates.
• Applying none. Sending raw text to an instruction-tuned model is more usually a cause of rambling than of silence, but on models trained with a strict format it can land in a region where the immediate continuation is a turn marker.
• Applying the wrong one. A GGUF converted without its template, or one where the converter substituted a generic default, inserts markers this model has never seen. Unknown markers tokenise into pieces, and the surrounding structure can still read as a finished turn.

The template is stored in the GGUF under a metadata key, tokenizer.chat_template. Print it and compare it against the model card’s documented format. In llama.cpp, --jinja makes the server use the embedded template rather than a built-in approximation, and several published models require it because their template uses control flow the built-in path does not implement.

Cause 2: the wrong EOS token in the file

GGUF records which token IDs terminate generation. If those IDs are wrong, behaviour goes wrong in one of two directions, and both have been widely reported on published conversions.

The famous case is Llama 3, where the instruct models end turns with <|eot_id|> (ID 128009) while some early conversions wrote the base model’s <|end_of_text|> (ID 128001) into the EOS field. That direction produces the opposite symptom — a model that never stops — but the same class of error in reverse, where a commonly-emitted token is marked as terminal, produces an immediate stop. Community conversions of newer models have shipped both kinds, and the fixes are published as corrected metadata rather than as new weights.

Check what the file claims:

# with the gguf python package installed
gguf-dump --no-tensors model.gguf | grep -i -e eos -e bos -e eot -e chat_template

# llama.cpp prints the same at load time
llama-cli -m model.gguf -p "hi" -n 8 2>&1 | grep -i -e "EOG token" -e "EOS token"

If the EOS ID does not match the model card, the file is wrong. Prefer a conversion from a publisher who has fixed it over patching metadata by hand; llama.cpp ships a script for rewriting GGUF key-value pairs, but a file whose metadata was wrong once is worth replacing.

Cause 3: nothing to do with the model

• The token budget is zero or tiny. A max_tokens of 0, or an -n 0, yields an empty string with no error. Some wrappers default it from a config file you have forgotten about.
• A stop string that matches immediately. Stop sequences are matched against generated text; an empty string, a newline, or a template marker in the stop list truncates before the first visible character. Anything containing a newline is worth suspecting.
• The context is already full. If the prompt fills the context window, there is no room to generate. The server usually warns, but a warning in a log is easy to miss and the API result is simply empty. Shorten the prompt or raise the context — see why the window and the output budget are two different limits.
• A grammar or JSON schema with no valid first token. Constrained decoding intersects the model’s distribution with what the grammar allows. An over-constrained grammar can leave only the stop token available.

Diagnosing it in one command

Do not change settings one at a time. Establish first whether the model can generate at all, by taking the chat layer out entirely:

1. Run a completion with no template: llama-cli -m model.gguf -p "The capital of France is" -n 20. If words appear, the weights and the runtime are fine and your problem is in the chat formatting.
2. Print what is actually being sent. In llama.cpp, verbose prompt output shows the exact tokenised prompt; in any wrapper, log the final string before it is tokenised. Look for a turn marker immediately before where the answer should begin, or for a marker appearing twice.
3. Compare that string to the model card’s documented format, character for character. Whitespace and newline placement inside these markers is part of the format.
4. If the string is right, dump the GGUF metadata and check the EOS and end-of-turn IDs against the card.
5. Only then look at sampling and stop strings — and clear the stop list entirely as a test rather than editing it.