A Local Model Won’t Stop Generating

The model answers your question, and then keeps going. Sometimes it prints something like <|im_end|> or <|eot_id|> as literal text and starts a new turn beginning User:, inventing both sides of a conversation. Sometimes there is no marker at all and it simply never returns. These are two different faults with two different fixes.

What you are looking at

The response, when it finally arrives, carries the evidence. An OpenAI-compatible endpoint reports "finish_reason": "stop" when generation ended because a stop condition fired, and "finish_reason": "length" when it ended because it hit the token cap. llama.cpp’s native endpoint reports the same distinction as a stop_type of eos, word or limit. If you are seeing length or limit, nothing stopped the model — a ceiling did. If you are seeing text that should have been a stop token rendered as visible characters, the model emitted the right token and the server did not recognise it as terminal.

Nothing was ever going to stop it

The reason this failure is so abrupt is that the defaults do not include a length limit. llama.cpp documents -n, --predict, --n-predict N as “number of tokens to predict (default: -1, -1 = infinity)”, and the same n_predict field in the JSON request body carries the same default. Ollama’s num_predict parameter is documented the same way: -1 means infinite generation. The stop array is documented as defaulting to an empty list.

So on a default configuration there are exactly two things that can end a generation: the model emitting a token the server has been told is terminal, or the context filling up. Both of those can fail, and when they do the loop has no other exit. This is a deliberate design — a completion server should not truncate your output because it guessed a limit — but it means the safety net you assume exists does not.

Telling the two causes apart

Send one request with a hard cap and read the fields rather than the prose:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Say hello in five words."}],
    "max_tokens": 64,
    "temperature": 0
  }' | jq '{finish: .choices[0].finish_reason, text: .choices[0].message.content}'

If finish_reason is stop and the text is five words, the stop token is recognised and your original problem was purely a missing cap. If it is length and the text has run past the answer into a fabricated next turn, the stop token is not being honoured, and capping the length only limits the damage. Confirm which token the server thinks is terminal by reading the model metadata at load time — llama.cpp prints the end-of-sequence and end-of-turn token ids and their string forms in the metadata block it logs while loading:

journalctl -u llama-server -b | grep -i -E 'eos|eot|chat template|BOS'

Cause one: the turn was never closed

An instruction-tuned model was trained on a specific turn structure: special tokens that open a role, the content, and a special token that closes it. The model has learned to emit the closing token because that is what always came next in training. Three things break that chain, and all three present identically.

• You bypassed the chat template. Posting a raw string to a text-completion endpoint sends exactly that string. The model never sees the opening role markers it was trained with, so it has no structure to close and continues as free text — usually by inventing a plausible dialogue. Using the chat-completions endpoint, or passing an explicit --chat-template, is what applies the structure.
• The GGUF metadata is wrong or missing. The terminal-token id lives in the file’s metadata, written during conversion. Conversions made before a tokenizer’s end-of-turn token was handled correctly — a well-documented problem with early Llama 3 conversions, where the model emits <|eot_id|> but the file declares a different end-of-sequence token — produce a model that stops correctly in its own view and not in the server’s. Re-downloading a current conversion from the publisher fixes this and hand-editing metadata should be the second choice.
• It is a base model. A non-instruct checkpoint has no turn structure at all and will continue for as long as you let it. Nothing is broken; you are using a text continuation engine to hold a conversation.

A fourth possibility worth ruling out in one second: --ignore-eos does exactly what it says and is documented as defaulting to false, but wrappers and front-ends do set it, sometimes under a friendlier name. If it is on, the server is discarding the stop token you are trying to make it honour.

The fix, in order

1. Always send a length cap. Not as a fix, as hygiene: it converts an unbounded failure into a bounded one, and it is the only defence that works when the cause is something you have not diagnosed yet. Set max_tokens (or n_predict, or num_predict) to a number that reflects the longest legitimate answer for your use, and treat finish_reason: length in your application as a condition to handle rather than a response to display.

2. Use the chat endpoint so the template is applied. This resolves the majority of cases on its own, because it hands the model the structure it was trained to close.

   # instead of POST /completion with a raw prompt string:
   POST /v1/chat/completions
   {"messages": [{"role":"user","content":"..."}], "max_tokens": 512}

3. Add explicit stop strings as a backstop. Stop strings are matched on the decoded text rather than on token ids, so they work even when the metadata does not. They are excluded from the returned completion, which is usually what you want.

   "stop": ["<|im_end|>", "<|eot_id|>", "\nUser:", "\n</s>"]

   Choose these to match the model family you loaded. A ChatML model closes with <|im_end|>; a Llama 3 family model uses <|eot_id|>. Adding all of them is harmless and the list costs nothing at runtime.

4. Verify the template the server is using. llama.cpp exposes the resolved chat template on its /props endpoint; compare it against the template on the model’s card. A mismatch here is the root cause hiding behind all of the above.

5. Check what happens at the context limit. If the generation is genuinely running until the context fills, the next thing that happens is truncation, and it may not be announced — silent context truncation covers what a server does when it runs out of room, which is the failure directly downstream of this one.

One symptom that looks like this and is not: output that stops being an answer and becomes the same phrase repeated indefinitely is degenerate repetition, a sampling failure rather than a stopping failure. The model is still perfectly willing to emit its stop token; it has fallen into a loop where the stop token is never the most likely continuation. The sampling settings page covers the parameters that govern it.