Qwen's Tokenizer Vocabulary Size

Ask what Qwen’s vocabulary size is and you can get 151,643, 151,646 or 151,936 depending on where you look. All three appear in the same repository, they mean different things, and only one of them is the one your code should use.

Three numbers, all correct

Pull any Qwen2, Qwen2.5 or Qwen3 repository and you can read each of these off a different file:

• 151,643 — the number of entries in vocab.json, the learned byte-level BPE merges and base tokens. This is the vocabulary in the linguistic sense: what the tokeniser learned from data.
• 151,646 and up — what len(tokenizer) returns, which adds the special tokens registered in added_tokens.json. For a Qwen2.5 instruct checkpoint that is <|endoftext|>, <|im_start|> and <|im_end|> at ids 151643–151645, plus a set of reserved control tokens for tool use and vision that Alibaba added at various points in the family. The exact total therefore differs between checkpoints.
• 151,936 — "vocab_size" in config.json, which is the width of the embedding matrix and of the output projection. This is the number that governs the shape of the logits tensor.

You can confirm all three against the repository directly — the Qwen team publishes them in the open on Hugging Face, for example at Qwen2.5-7B-Instruct’s config.json. The stability of 151,936 across Qwen2, Qwen2.5 and Qwen3 is itself useful information: the tokeniser did not change across those generations, so token counts computed for one are valid for the others, and a cost estimate built on Qwen2.5 does not need redoing for Qwen3.

Why the embedding matrix is bigger

151,936 − 151,643 = 293, and there are not 293 special tokens. The remainder is padding, and it is there for a hardware reason rather than a linguistic one.

The final matrix multiplication in a forward pass projects the hidden state onto the vocabulary. Its second dimension is the vocabulary size, and GPU matrix-multiply kernels are substantially faster when that dimension is a multiple of a large power of two, because the tile sizes the kernel decomposes the problem into then divide it evenly. 151,936 is 128 × 1,187 — a clean multiple of 128, where 151,646 is not. Padding to the next such multiple costs a few megabytes of unused embedding weights and buys a measurable improvement in kernel occupancy on the single largest matmul in the model.

The padded rows are dead. They are initialised, never trained to anything meaningful, and correspond to no token, which has a practical consequence: if you ever sample directly from the logits without masking, ids above the real vocabulary size can in principle be drawn and will decode to nothing. Every standard runtime handles this; a hand-rolled sampler might not.

What the vocabulary is made of

Qwen uses byte-level BPE, the same class of tokeniser as GPT-4o’s and Llama 3’s. Two properties follow from “byte-level” and they are the ones that matter operationally.

First, there is no out-of-vocabulary case. Every possible byte sequence encodes, because the 256 single-byte tokens are always in the vocabulary as a fallback. A tokeniser cannot fail on unusual input; the worst case is that it encodes badly, at one token per byte.

Second, the efficiency of the encoding on a given script depends entirely on how much of that script was in the corpus the merges were learned from. Qwen was built as a multilingual model with substantial Chinese training data, and a large part of its 151k vocabulary is spent on CJK sequences that a predominantly-English tokeniser would encode byte by byte. That is why the vocabulary is as large as it is: covering two writing systems well requires more entries than covering one.

The direct consequence for anyone billing by the token: identical Chinese text costs dramatically fewer tokens under Qwen’s tokeniser than under a tokeniser without that coverage, and identical English text costs marginally more, because vocabulary spent on CJK is vocabulary not spent on English word fragments. Which way that trade lands for you is a question about your corpus, not about the models.

Measuring it against Llama’s tokeniser

The honest way to answer “how many more tokens will this cost under Qwen” is to run both tokenisers over your own text. Any single published ratio is a statement about somebody else’s sample. This takes about a minute:

from transformers import AutoTokenizer

qwen  = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
llama = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")

with open("sample.txt", encoding="utf-8") as f:
    text = f.read()

q = len(qwen(text)["input_ids"])
l = len(llama(text)["input_ids"])

print(f"qwen  {q}")
print(f"llama {l}")
print(f"ratio {q / l:.3f}")   # >1 means Qwen needs more tokens for this text

Use a real sample — a few thousand words of the traffic you actually send, not a paragraph of Wikipedia. Prompt text with heavy formatting, JSON, code or non-Latin script diverges from prose far more than prose diverges from prose, and the ratio you get on a well-behaved English paragraph will not survive contact with a system prompt full of nested JSON schemas. Compare with Llama 3’s vocabulary size for the other side of that ratio.

What the size actually changes

• Your cost model, if you switch families. Prices are quoted per token and tokens are not a common unit across tokenisers. A model that is 10% cheaper per token and needs 15% more tokens for your text is more expensive. This is the only place the vocabulary size touches your bill, and it touches it every request.
• How much fits in the context window. A 32,768-token window holds a different amount of text per family, and the difference is largest exactly where the vocabularies differ most — non-Latin script and code.
• Almost nothing about quality. A larger vocabulary means shorter sequences for covered scripts, which means fewer forward passes per unit of text and slightly cheaper long-range attention. It does not make a model better at reasoning, and the vocabulary size is a poor proxy for anything you care about beyond encoding efficiency.