Llama 3’s Tokenizer Vocabulary Size, and What Changed From Llama 2

Llama 2 tokenizes with a 32,000-token vocabulary. Llama 3 tokenizes with 128,256. That is a four-fold increase between two consecutive releases of the same family, and it is the change behind most of the “why does my token count not match any more” confusion.

The two numbers

Llama 2   32,000 tokens    SentencePiece BPE
Llama 3  128,256 tokens    tiktoken-style BPE
                           = 128,000 learned + 256 reserved

The 128,256 figure splits cleanly and it is worth knowing why. 128,000 ids are learned merges. The 256 above them are reserved slots holding the chat-format tokens — <|begin_of_text|> at 128000, <|eot_id|> at 128009 and the rest — plus a run of placeholders that later releases claimed for new purposes without changing the vocabulary size. That is how Llama 3.1 could add <|eom_id|> and <|python_tag|> and Llama 3.2 could add <|image|> while remaining tokenizer-compatible with Llama 3. The reserved block was designed for exactly that.

The whole vocabulary is enumerated in the tokenizer files shipped with each checkpoint, and the reserved ids are listed in Meta’s release documentation in the meta-llama/llama-models repository.

SentencePiece out, tiktoken BPE in

The size is the headline, but the implementation swap underneath it is what changes behaviour. Llama 2 used a SentencePiece model, which treats the input as a stream of Unicode characters, marks word boundaries with a visible metacharacter, and by default prepends a space to the input. Llama 3 uses a byte-level BPE of the kind popularised by tiktoken, which operates on UTF-8 bytes with a regex pre-tokenizer that splits contractions, digit runs and whitespace before merges are applied.

Three practical consequences follow from that swap alone, independent of the size change:

• No implicit leading space. Llama 2’s tokenizer added one; Llama 3’s does not. Code that stripped the resulting artefact will now strip a real character.
• Digits group predictably. The byte-level pre-tokenizer splits long numbers into fixed-width chunks rather than into whatever merges happened to be learned, which makes arithmetic behaviour less erratic than it was in Llama 2.
• Every byte sequence is representable. Byte-level BPE has no unknown token, so no input can fail to encode. Llama 2’s vocabulary had byte fallback for the same reason, but it spent tokens doing it.

The size change and the implementation change compound on non-English text, which is where the difference is largest. A 32,000-token vocabulary trained predominantly on English has very few merges to spend on other scripts, so words in those languages decompose into many short pieces — sometimes into individual bytes, which for a non-Latin script can mean two or three tokens per character. Quadrupling the vocabulary buys room for merges those languages can actually use. That is why any average token-count reduction understates the effect for some users and overstates it badly for others, and why the only figure worth acting on is the one from your own corpus. The per-tokenizer differences are the subject of token count differences across Llama tokenizers.

What a bigger vocabulary costs

A larger vocabulary is not free, and the cost lands in a specific place — the embedding matrices. This is arithmetic, not a measurement, and the assumptions are stated so you can check it.

Assumptions: hidden size 4,096 for both Llama 2 7B and Llama 3 8B, as given in their configs; separate input and output embedding matrices (untied), so the vocabulary is paid for twice; parameters counted, not bytes.

Llama 2 7B
  32,000 x 4,096          =   131.1M per matrix
  x 2 (input + output)    =   262.1M parameters

Llama 3 8B
  128,256 x 4,096         =   525.3M per matrix
  x 2 (input + output)    = 1,050.6M parameters

Difference                =   788.5M parameters

That single change accounts for most of the gap between a “7B” Llama 2 and an “8B” Llama 3. The transformer body barely grew; the vocabulary did. It also explains why the smaller Llama 3.2 models feel disproportionately embedding-heavy: a 1B model carrying the same 128,256-row matrices is spending a large fraction of its parameters on the vocabulary, which is part of why those models tie their input and output embeddings.

Reproducing the comparison

Meta stated in its Llama 3 announcement of 18 April 2024 that the new tokenizer encodes language more efficiently, giving a figure of up to 15% fewer tokens. “Up to” is doing real work in that sentence: the gain depends heavily on the language and the content, and it is largest on non-English text and on text with long common words. Rather than quote a number for your text, run it:

1. Accept the licence for both checkpoints on Hugging Face. Both repositories are gated and the download will fail with a 401 until you have.
2. Load only the tokenizers. You do not need the weights, which is why this runs on a laptop in seconds.
3. Encode the same string with both and compare lengths, using text that resembles your actual workload rather than a sentence chosen to flatter one of them.

from transformers import AutoTokenizer

t2 = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
t3 = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

for text in [
    "The quick brown fox jumps over the lazy dog.",
    "def normalise(rows): return [r.strip().lower() for r in rows]",
    "Le renard brun rapide saute par-dessus le chien paresseux.",
]:
    a = len(t2.encode(text, add_special_tokens=False))
    b = len(t3.encode(text, add_special_tokens=False))
    print(f"{a:4d}  {b:4d}  {1 - b / a:+.1%}  {text[:40]}")

add_special_tokens=False matters here. Left at its default, each tokenizer adds its own BOS and the comparison is off by one in a direction that varies.

Adding tokens, and why not to

Fine-tuners regularly want a new special token — a delimiter for a custom format, a marker for retrieved passages, a role that does not exist in the reference template. There are two ways to get one and they have very different consequences.

The route people reach for first is tokenizer.add_special_tokens() followed by model.resize_token_embeddings(). That grows the vocabulary past 128,256, which means the embedding matrices are reallocated, the new rows start untrained, and — the part that bites later — the checkpoint no longer matches the tensor shapes any other tool expects. Quantisers, GGUF converters, LoRA adapters trained against the original shape, and serving stacks that validate the config against a known vocabulary size all have to be told. It works, and it is a long tail of small incompatibilities.

The route Meta designed for is the reserved block. There are roughly 240 unused ids named <|reserved_special_token_N|> sitting inside the existing 128,256, and repurposing one is a rename in the tokenizer files with no shape change anywhere. The embedding row is untrained either way — that is unavoidable for a genuinely new token — but nothing downstream has to know. Meta used exactly this mechanism itself to add <|eom_id|> in 3.1 and <|image|> in 3.2, which is a reasonable endorsement of the approach.

The third option, and often the best one, is to use no new token at all and delimit with an ordinary string the tokenizer already handles well. A few extra tokens per delimiter is usually cheaper than a checkpoint that is subtly non-standard.

What actually changes for you

• Old token estimates are wrong. A character-per-token ratio calibrated against Llama 2 overestimates Llama 3 counts. If you budget context or bill by tokens from a stored estimate, it needs recomputing. See the tokens-per-word estimate for Llama 3.
• The effective context grew more than the number suggests. Fewer tokens per unit of text means an 8,192-token Llama 3 window holds noticeably more English than a 4,096-token Llama 2 one — more than the 2× the numbers imply.
• Cross-family token counts are not comparable. A token count from a Llama tokenizer does not predict the count from OpenAI’s o200k_base or from Anthropic’s. Count with the tokenizer of the model you are about to call, not with the one you happen to have imported.
• Fine-tuning data must be re-tokenized. A cached tokenized dataset from a Llama 2 project is not reusable. The ids mean different things.