DeepSeek's Tokenizer and Vocabulary Size

DeepSeek-V3 ships a byte-level BPE tokenizer with a vocabulary in the 129,000 range. The number itself decides almost nothing; what it is near, and which special tokens are in it, decides quite a lot.

The number and where to read it

The authoritative value is vocab_size in the model repository’s config.json, and for DeepSeek-V3 it is 129,280. Read it from the repository rather than from any article, including this one, because it is a per-checkpoint property: earlier DeepSeek generations shipped a materially smaller vocabulary, and a future one may ship a different figure again.

python -c "import json,urllib.request as u; \
print(json.load(u.urlopen('https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json'))['vocab_size'])"

There is a second number, and confusing the two causes real bugs. The model’s embedding matrix is sized by vocab_size, but the tokenizer’s own len(tokenizer) counts the entries actually defined, including added special tokens, and the two need not be equal. Model configs are commonly rounded up to a convenient multiple for hardware reasons, leaving a handful of embedding rows that no token ever maps to. If you are fine-tuning and adding tokens, check both before resizing anything.

“Byte-level BPE” is the other half of the answer. It means the tokenizer’s alphabet is the 256 possible bytes, so there is no input it cannot encode — no unknown token, no failure on unusual scripts, emoji or binary-looking strings. The worst case is that something unfamiliar encodes to one token per byte, which is expensive but never an error.

The special tokens

DeepSeek’s role markers are visually distinctive and are the fastest way to identify a DeepSeek prompt at a glance. They use full-width vertical bars and a middle dot as separators, producing tokens of the form <|begin of sentence|>, <|User|>, <|Assistant|> and <|end of sentence|> — with the actual characters being the full-width bar rather than the ASCII pipe, and word separators being a middle dot rather than a space.

That detail matters the moment you hand-build a prompt string. Typing the ASCII-looking approximation gets you a sequence of ordinary text tokens instead of the single control token the model was trained on, and the model will not treat it as a turn boundary. The result is a model that rambles past where it should have stopped, or ignores the conversation structure entirely. Copy these tokens from tokenizer_config.json or, better, never write them by hand — use apply_chat_template.

The reasoning checkpoints add the thinking delimiters to this set, and the template emits an opening one as part of the generation prompt. That is why locally served R1 produces literal tags while the hosted API does not: parsing the trace is entirely a consequence of what these tokens do.

Comparing with another tokenizer

A vocabulary size near 129,000 puts DeepSeek in the same broad class as Llama 3, whose tokenizer Meta documents at 128,256 entries — and both are roughly four times the 32,000-entry vocabularies of the previous generation. Larger vocabularies mean fewer tokens for the same text, which is why per-token comparisons between model families are only meaningful once you have counted your own text with both.

That comparison is worth doing rather than reading about, because the answer depends on what your text is: two tokenizers of similar size can differ by a wide margin on code, on non-English languages, or on strings full of punctuation, while agreeing almost exactly on English prose. Here is the script.

from transformers import AutoTokenizer

samples = {
    "english":  "The quick brown fox jumps over the lazy dog.",
    "code":     "def merge(a: list[int], b: list[int]) -> list[int]:\n    return sorted(a + b)",
    "json":     '{"user":{"id":42,"tags":["a","b"],"active":true}}',
    "dutch":    "De vergadering is verplaatst naar volgende week donderdag.",
}

tokenizers = {
    "deepseek-v3": AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3"),
    "llama-3.1":   AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B"),
}

print(f"{'sample':10} " + " ".join(f"{n:>12}" for n in tokenizers))
for name, text in samples.items():
    counts = [len(t.encode(text, add_special_tokens=False)) for t in tokenizers.values()]
    print(f"{name:10} " + " ".join(f"{c:>12}" for c in counts))

Run it on your own corpus, not on those four lines. The number you actually want is tokens per thousand characters of your traffic, because that is what converts a published per-token price into a cost you can forecast. Note that the Llama repository is gated and requires accepting Meta’s licence before the download will succeed.

DeepSeek's own rule of thumb

For sizing rather than billing, DeepSeek publishes a per-character approximation in its API documentation: roughly 0.3 tokens per English character and roughly 0.6 tokens per Chinese character. Those are stated as rough figures for estimation, and DeepSeek says so — the exact count comes from the tokenizer.

The English figure translates to something in the region of one token per three or four characters, which is the same neighbourhood as most modern tokenizers on English prose. The Chinese figure is the more interesting one: at roughly 0.6 tokens per character, Chinese text is about twice as token-dense per character as English but far denser in meaning per character, so a Chinese document usually costs fewer tokens than its English translation. That is a direct consequence of DeepSeek having trained the tokenizer on a corpus where Chinese is heavily represented.

Counting exactly

Three ways to get the true count, in increasing order of trust.

• The published tokenizer package. DeepSeek distributes a tokenizer alongside its token-usage documentation for exactly this purpose. Loading it locally costs nothing per call and is the right tool for pre-flight checks and budget enforcement.
• The model repository’s tokenizer. AutoTokenizer.from_pretrained on the checkpoint you are actually serving. This is the correct source when you self-host, because it is the same file the server uses.
• The usage object. Authoritative by definition, because it is what you are billed on — and it will be slightly higher than your local count, because it includes the chat-template scaffolding your local encode of a bare string did not. Reconcile against it; do not be surprised by the gap.

If you need the local and the billed number to agree, tokenize the output of apply_chat_template rather than the raw message text. That closes most of the difference, and what remains is a small constant per request.

Why the size matters at all

Vocabulary size is not a quality metric and a larger one is not better. It is a tradeoff with effects at both ends of the model, and knowing which effects are real stops it being a number people quote at each other.

• Fewer tokens for the same text. A larger vocabulary can afford longer merged units, so common words and code idioms become single tokens instead of two or three. Same text, fewer tokens: cheaper per request at a given price, faster to generate, and more of it fits in a fixed context window.
• A larger embedding and output layer. Every entry needs an embedding row and a column in the final projection. Growing the vocabulary grows both, which costs memory and adds arithmetic to the step that produces the distribution over next tokens. This is the cost side and it is why vocabularies are not simply made enormous.
• Coverage decides who pays. A tokenizer trained largely on English encodes other languages inefficiently, sometimes at close to one token per character. That is a direct cost and context penalty for those users, and it is a real reason to check your own languages rather than trusting an average.
• Per-token prices are not comparable across families. Two providers quoting the same price per million tokens can differ materially in cost for identical text, because their tokenizers disagree about how many tokens that text is. Convert to cost per thousand characters of your own traffic before comparing anything.

The last point is the one with money attached, and it is the reason the comparison script above is worth running once on a representative sample. It converts a published rate into a number you can actually use, and the conversion factor is different for every model family and every corpus.