Llama 3’s Chat Template and Special Tokens

Llama 3 replaced Llama 2’s bracket-and-tag prompt format with one built entirely out of reserved tokens. The change is worth understanding character by character, because the format has no tolerance: a missing newline is a different prompt, and the model will answer it.

The special token inventory

Llama 3’s tokenizer reserves 256 ids at the top of its vocabulary. Six of them carry the chat format, and their ids are stable across the 3.x line:

128000  <|begin_of_text|>       start of the whole sequence
128001  <|end_of_text|>         end of generation (base model EOS)
128006  <|start_header_id|>     opens a role header
128007  <|end_header_id|>       closes a role header
128008  <|eom_id|>              end of message (3.1+, tool calls)
128009  <|eot_id|>              end of turn (instruct model)

Two more appear in 3.1 and later: <|python_tag|>, which prefixes a built-in tool call, and <|finetune_right_pad_id|>, which exists so that fine-tuners have a padding token that is not <|end_of_text|>. Llama 3.2 Vision adds <|image|>. The remaining reserved ids are placeholders named <|reserved_special_token_N|>, and 128008 was one of them in the original Llama 3 before 3.1 gave it a meaning.

The pair to keep straight is the last two. <|eot_id|> ends a conversational turn; <|end_of_text|> ends the document. The instruct models emit the first and the base models emit the second, and checking for only one of them is a common and quiet bug — the two end tokens have a page of their own.

A rendered conversation, byte for byte

Here is a two-turn exchange in the format Meta documents, with a system message and one completed assistant turn. Every character below is part of the prompt.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a terse assistant. Answer in one sentence.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the capital of Peru?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Lima.<|eot_id|><|start_header_id|>user<|end_header_id|>

And its population?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Reading it as a structure rather than a wall of tokens:

• <|begin_of_text|> appears exactly once, at the very start. It is not repeated per turn.
• Each message is <|start_header_id|>ROLE<|end_header_id|>\n\nCONTENT<|eot_id|>, with no space between the tokens and the role name.
• The valid roles are system, user, assistant, and — from 3.1 — ipython for returning tool results. They are lowercase.
• The prompt ends with an open assistant header and its two newlines, and nothing after them. That trailing fragment is what tells the model it is its turn to speak.

Meta publishes this format, with the tool-calling variants, in the prompt-format documentation at llama.com’s Llama 3.1 model card and prompt formats page.

The two newlines are part of the format

After <|end_header_id|> there are exactly two newline characters before the content begins. This is the single most common hand-rolling mistake, and it is invisible in a terminal.

It matters because the model was fine-tuned on this byte sequence. The tokenizer encodes \n\n after a header differently from a single newline or from a newline plus a space, so a prompt with one newline is a sequence the instruct tuning never saw. The usual symptom is not a crash: it is an answer that is subtly worse, or a model that starts its reply with a stray newline, or a model that ignores the system message. Degradation without error is exactly the failure mode that survives a code review.

Equally, there is no newline before <|start_header_id|>. The <|eot_id|> of one message butts directly against the header token of the next. If you find yourself joining messages with "\n".join(...), the format is already wrong.

There is a reason the format is this unforgiving, and it is not fastidiousness. The tokenizer is a byte-level BPE, so whitespace is not a separator it discards — it is content that participates in merges. A newline followed by a capital letter can encode to a different token than a newline followed by a space and the same letter, which means two strings that look identical in a log are different sequences of integers by the time the model sees them. Nothing normalises between your string and the forward pass, so there is no layer that could forgive the difference even in principle.

Applying it without writing it by hand

The correct answer to all of this is to not build the string yourself. The tokenizer ships the template, and apply_chat_template renders it:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user", "content": "What is the capital of Peru?"},
]

prompt = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,   # appends the open assistant header
)
print(repr(prompt))

add_generation_prompt=True is the flag that appends the trailing <|start_header_id|>assistant<|end_header_id|>\n\n. Leave it out and the model will happily continue the user’s message instead of replying to it. Printing with repr rather than print is deliberate: it is the only way to see whether the newlines are there.

One thing the rendered output will show that you did not put there: the Llama 3.1 template injects a Cutting Knowledge Date and a Today Date line into the system block, even when you supplied no system message. That behaviour belongs to the system block specifically and is covered in how the reference template structures the system role.

Where the template physically lives

“The Llama 3 chat template” is not one artefact, and knowing which copy your stack is reading is most of debugging it. There are four, and they can disagree.

• Meta’s published reference — prose and examples on llama.com and in the model cards. Authoritative about intent, not consulted by any software.
• The chat_template field in tokenizer_config.json — a Jinja string shipped with the Hugging Face checkpoint. This is what apply_chat_template executes, and it is the one that injects the date lines. It is per-checkpoint, so a fine-tune can ship a modified one without saying so anywhere else.
• The tokenizer.chat_template metadata key inside a GGUF file — copied in at conversion time by the conversion script. If the script was older than the checkpoint, this copy is stale, and it is the origin of a large share of “works in Transformers, behaves oddly in llama.cpp” reports.
• A TEMPLATE block in an Ollama Modelfile, or the equivalent in a serving stack’s config — a hand-written override that takes precedence over all of the above. Overriding the template and forgetting to carry the stop parameters forward is a specific, common, and completely silent way to break a model.

When output looks subtly wrong, print the rendered prompt from the layer closest to the model rather than reasoning about which template should be in effect. Every one of these systems can show you the final string, and the final string is the only thing the model sees.

What going wrong looks like

• Literal <|eot_id|> in the output. The special tokens were encoded as ordinary text. Something in the chain passed the rendered string through a tokenizer call with add_special_tokens handling that did not recognise them, or through a completion endpoint that escapes them by policy.
• Two <|begin_of_text|> tokens. You rendered the template to a string and then tokenized it with the default add_special_tokens=True, which prepends BOS again. Use tokenize=True on the template call, or pass add_special_tokens=False.
• The model answers as the user. The generation prompt is missing, or the last message in your list already had a closing <|eot_id|> and nothing followed it.
• The model keeps going after its answer. The stop token was not registered with the sampler. On raw generate() calls this needs an explicit eos_token_id list.