Llama 3’s Two End Tokens: eot_id and end_of_text

Llama 3 has two tokens that mean “stop”, and which one you get depends on whether you loaded a base checkpoint or an instruct one. Almost every “why does my Llama keep talking” report is this, and it is a two-line fix once you know which line.

Two tokens, two meanings

128001  <|end_of_text|>   end of the document
128009  <|eot_id|>        end of this conversational turn

<|end_of_text|> is the classic EOS token of a language model. It marks the end of a training document. A base Llama 3 checkpoint — the pretrained model, no instruct tuning — emits it when it decides the text is finished, and that is the only stop signal it has.

<|eot_id|> was added for the chat format. It means “I have finished my turn, control passes back”, and the instruct checkpoints were fine-tuned to emit it at the end of every assistant reply. Look at a rendered conversation and you will see it closing every message, user and assistant alike; it is the message terminator of the format described in the chat template page.

The distinction is real: one turn ending is not the conversation ending. An instruct model can still emit <|end_of_text|> — the token was not removed from its vocabulary and it retains the pretraining behaviour underneath the tuning — so both are live for an instruct model and only the first is live for a base one. That asymmetry is why every correct configuration lists both rather than picking one.

There is a reason it was done this way rather than by reusing the existing EOS token for turns. During instruct tuning the model is trained on multi-turn conversations, and a single sequence contains several turn boundaries and one document boundary. If the same token marked both, the model would have no way to learn the difference between “this reply is finished” and “this conversation is finished” — and, more practically, the training data could not distinguish a turn boundary from the end of an example. Llama 2 solved the same problem with [/INST] text markers, which worked but consumed ordinary tokens and could be produced accidentally by a model quoting its own format. A reserved id cannot be emitted by accident from ordinary text, because no sequence of characters encodes to it.

What checking only one looks like

Configure an instruct model with eos_token_id = 128001 alone, which is the value in the tokenizer’s eos_token field and therefore the value a naive setup inherits, and you get output like this:

Lima is the capital of Peru.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is its population?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Lima's metropolitan area has around 10 million people.<|eot_id|>...

The model answered correctly, emitted its turn terminator, and then — because nothing stopped it — carried on generating the rest of the conversation. It writes the user’s next question, answers that too, and keeps going until it hits max_new_tokens. You are billed for all of it, and the fabricated user turn is the model’s invention rather than anything a user said.

The symptom has two faces depending on your decoder. If special tokens are being skipped on decode, you do not see the <|eot_id|> markers at all and just get a rambling monologue that appears to answer questions nobody asked. If they are not skipped, you see the tokens literally, which is at least a diagnostic. Set skip_special_tokens=False while debugging for exactly this reason.

The generation config

For the instruct checkpoints, Meta ships a generation_config.json that already lists both ids. The correct thing is to let it be loaded rather than to override it:

{
  "bos_token_id": 128000,
  "eos_token_id": [128001, 128009],
  "do_sample": true,
  "temperature": 0.6,
  "top_p": 0.9
}

Note that eos_token_id is a list. Code written against a model where it was a single integer — which is most code written before Llama 3 — will silently take the first element or fail a type check. If you are constructing the call by hand, pass the list explicitly:

terminators = [
    tok.eos_token_id,                          # 128001
    tok.convert_tokens_to_ids("<|eot_id|>"),   # 128009
]

out = model.generate(
    **inputs,
    max_new_tokens=512,
    eos_token_id=terminators,
)

Looking the id up by name rather than hard-coding 128009 is worth the extra line. The numeric ids are stable across the 3.x line, but a fine-tune that added tokens is not obliged to keep them.

Runtime by runtime

• Transformers. Handled if generation_config.json is present and you do not override eos_token_id. Overriding it with a single value is the usual way this breaks.
• vLLM. Reads the generation config and stops on both. Adding stop=["<|eot_id|>"] to your sampling params is harmless belt-and-braces, though note that a stop string is matched on decoded text and a stop token is matched on the id, which is the more reliable of the two.
• llama.cpp and GGUF. Depends on the metadata baked in at conversion time. Older GGUF conversions of Llama 3 were widely reported with only <|end_of_text|> as EOS, which is the origin of most of the community reports of this bug. A re-quantised file from a current converter fixes it; so does passing the stop token explicitly.
• Ollama. Set in the Modelfile’s PARAMETER stop lines. A custom Modelfile that overrides the template without carrying the stop parameters forward reintroduces the problem.

If you have set all of this correctly and generation still runs on, the next thing to check is whether your stop strings are being matched across a token boundary in a streaming decoder — a different problem, covered in why Llama appears to ignore a stop sequence.

Check which checkpoint you loaded

Before chasing configuration, rule out the simpler explanation: a base checkpoint will never emit <|eot_id|> at all, because it was never tuned to. No amount of stop-token configuration fixes a base model being asked to behave like a chat model — it will continue your text, plausibly and indefinitely, until it emits <|end_of_text|> or runs out of budget.

The repository name is the usual tell — Llama-3.1-8B is the base model and Llama-3.1-8B-Instruct is the tuned one — but names get changed on re-uploads and quantised copies are inconsistently labelled. Two checks that do not depend on the name:

# 1. Does the tokenizer ship a chat template?
python -c "from transformers import AutoTokenizer as T; \
t=T.from_pretrained('.'); print(bool(t.chat_template))"
# base: False (or a generic fallback)   instruct: True

# 2. Does the generation config list two EOS ids?
python -c "import json; print(json.load(
open('generation_config.json')).get('eos_token_id'))"
# base: 128001        instruct: [128001, 128009]

The second check is the more useful of the two, because it tells you both which checkpoint you have and whether its shipped configuration is correct. A checkpoint advertised as instruct whose generation config reports a bare 128001 has been repackaged by somebody who dropped the list, and that single value is the entire bug.

And in 3.1, a third

Llama 3.1 gave meaning to id 128008 as <|eom_id|>, “end of message”, which terminates an assistant message that expects a tool result rather than a user reply. It is emitted only when the system block sets Environment: ipython, so most deployments never see it — but a tool-using one that does not treat it as a terminator will generate straight past the tool call and hallucinate the result. The details are in the function calling format.