Why Llama 3's Token Counts Differ From Llama 2's on Identical Text

Move a prompt from Llama 2 to Llama 3 and the token count drops without a character changing. Nothing was compressed. The two models use unrelated tokenizers with vocabularies four times apart in size, and the bigger one simply needs fewer pieces to spell the same words.

Two different tokenizers, not one tuned

Llama 2 used a SentencePiece byte-pair-encoding model with a vocabulary of 32,000 pieces. Llama 3 replaced it wholesale with a byte-level BPE tokenizer in the tiktoken style, with 128,256 entries — 128,000 learned tokens plus 256 reserved special ones, which is where <|begin_of_text|>, <|eot_id|> and the rest of the chat template tokens live. Meta describes the change in its Llama 3 announcement of April 2024 and the vocabulary size is visible in the checkpoint itself, as vocab_size in config.json.

Because these are different vocabularies, token IDs do not correspond between them at all. ID 15043 means one thing to Llama 2 and something unrelated to Llama 3. Anything you cached as IDs — a pre-tokenized dataset, a stored prompt, a list of banned token IDs, a stop_token_ids array in a serving config — is invalid across the jump and will fail silently rather than loudly, because every ID in range is a legal ID.

One property both share is worth naming, because it removes a distraction: neither has an unknown-token problem. Both fall back to bytes for anything they cannot represent, so any input encodes to something — there is no <unk> and no failure mode where text is lost. The difference between them is entirely about how many tokens the encoding takes, never about whether it succeeds. That is why the symptom of a tokenizer mismatch is a cost or a length surprise rather than an error.

Why a bigger vocabulary means fewer tokens

A BPE tokenizer is trained by repeatedly merging the most frequent adjacent pair in a corpus, and the vocabulary size is the budget for how many merges it keeps. With 32,000 slots the budget runs out while plenty of common English word forms are still unmerged, so they arrive at inference as two or three pieces. With 128,000 slots the same training run keeps going: whole words, common suffixes, frequent bigrams and a great deal of code punctuation each earn their own entry.

Three specific consequences account for most of the observed gap:

• Whole-word coverage. Longer and less common English words that Llama 2 splits — technical vocabulary especially — are more often single tokens in Llama 3.
• Digits. Llama 2’s tokenizer splits every digit into its own token, so 2026 is four tokens. Llama 3’s pre-tokenization regex groups digits in runs of up to three, so the same number is one or two. Anything numeric — logs, tables, JSON with IDs, financial text — shows the largest improvement of all.
• Non-English text. A 32,000-piece vocabulary trained largely on English falls back to near-byte-level fragments for other scripts. The larger vocabulary affords real coverage, and languages outside Latin script benefit far more than English does.

There is a cost on the other side of the ledger, which is why nobody simply uses a million-token vocabulary: the embedding matrix and the output projection both scale with vocabulary size, so a 128K vocabulary adds parameters and adds work to the final softmax of every forward pass. Meta’s judgement was that the shorter sequences pay for it, because attention cost grows with the square of sequence length while the vocabulary cost grows linearly.

How large the difference is

In its April 2024 announcement of Llama 3, Meta stated that the new tokenizer encodes language more efficiently, giving up to around 15% fewer tokens. That is a published upper figure from the model’s author, and it is the only number on this page that came from anywhere other than arithmetic.

Where the difference bites

Four places, in rough order of how expensive the surprise is.

• Context budgets. A prompt builder tuned to fill a Llama 2 window will underfill a Llama 3 one. That is the harmless direction. The harmful direction is a length estimate carried forward from a Llama 3 measurement to a model with a smaller vocabulary, which overflows.
• Cost estimates. Per-token prices are not comparable across models that count tokens differently. Two providers quoting the same price per million tokens are not quoting the same price per document if their tokenizers differ.
• max_tokens for output. An output cap is in tokens, so the same cap buys more text on the larger vocabulary. A limit tuned on Llama 2 to produce roughly a paragraph produces a longer one on Llama 3.
• Chunking for retrieval. Chunk sizes expressed in tokens change meaning when the tokenizer changes, which quietly alters retrieval quality in a pipeline that otherwise looks untouched.

Migrating a pipeline across the change

A move from a Llama 2 derivative to a Llama 3 one is usually described as a weight swap. The tokenizer change makes it more than that, and the list of things to re-derive is short but nothing on it is optional:

1. Re-tokenize anything stored as IDs. Cached tokenizations, pre-tokenized training sets, stored prompt prefixes. Old IDs are still valid IDs, so nothing will error — you will simply be feeding the model a different text than you think.
2. Replace every hardcoded special-token ID. Llama 2’s end-of-sequence ID does not exist as such in Llama 3, whose Instruct models stop on <|eot_id|>. A stale stop_token_ids list is one of the more confusing ways to get a model that never stops.
3. Re-derive chunk sizes and truncation limits. Anything expressed in tokens now means a different amount of text. Retrieval chunking is the case that degrades silently, because nothing fails — the chunks are just a different size than the pipeline was tuned for.
4. Re-run the token-count estimates behind your costs. Per-request token counts fall, which is good, but any capacity plan or budget built on the old numbers is now wrong in both directions: fewer tokens per document, and a different price per token.
5. Rewrite the prompt template. Llama 2’s [INST] and <<SYS>> markers are plain text to a Llama 3 tokenizer — they tokenize into ordinary pieces and mean nothing to the model. Use the shipped chat template rather than porting the string.

Counting it on your own text

Both tokenizers load from the Hub, so the comparison is a dozen lines. Note add_special_tokens=False — otherwise you are also counting the template tokens each model adds, which is a different question.

from transformers import AutoTokenizer

text = open("sample.txt", encoding="utf-8").read()

llama2 = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
llama3 = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

n2 = len(llama2.encode(text, add_special_tokens=False))
n3 = len(llama3.encode(text, add_special_tokens=False))

print(f"llama 2: {n2} tokens")
print(f"llama 3: {n3} tokens")
print(f"difference: {(n2 - n3) / n2:.1%} fewer")

Both repositories are gated, so you need an accepted licence and a token in the environment before the download works. Run it over a real sample of your own traffic rather than a paragraph of clean prose — the answer for your workload is the only one that matters, and it is the one no article can give you.

If what you actually want is a rule of thumb for fitting documents into a window rather than a comparison, the tokens-per-word arithmetic is the page for that.