Gemma's Tokenizer: a 256K Vocabulary Shared With Gemini

Gemma’s tokenizer has around a quarter of a million entries. For a 2B-parameter model that is a strange-looking choice, because the embedding table alone then accounts for a large fraction of the parameter count. It is a deliberate inheritance, and it changes what your prompts cost.

The size, and how to read it off the model

Gemma 1 and Gemma 2 use a SentencePiece vocabulary of 256,000 pieces, with the embedding matrix padded to 256,128 rows so it divides evenly for sharding. Gemma 3 moved to a larger vocabulary of 262,144 entries, taken from a newer Gemini tokenizer with better coverage of non-English text. Both numbers are in the checkpoints:

from transformers import AutoConfig, AutoTokenizer

for repo in ["google/gemma-2-9b-it", "google/gemma-3-4b-it"]:
    cfg = AutoConfig.from_pretrained(repo)
    tok = AutoTokenizer.from_pretrained(repo)
    print(repo, cfg.vocab_size, len(tok))

The two printed numbers can differ by a few entries, and that is not a bug. vocab_size is the width of the embedding matrix, which is padded; len(tok) is how many pieces the tokenizer knows about, including the special tokens. Code that assumes they are equal breaks on exactly the models where the padding exists. Google documents the vocabulary in the Gemma technical reports linked from ai.google.dev/gemma/docs.

The Gemini lineage

Google’s technical reports describe Gemma’s tokenizer as a subset of the SentencePiece tokenizer used for Gemini. That is a sentence worth reading carefully, because it explains several things at once.

It means the segmentation behaviour of an open 2B model and a frontier closed model are close relatives: the same merges, the same treatment of scripts, the same handling of code. It means a token count you measure locally with Gemma’s tokenizer is a better estimate for Gemini than a count from any other family would be, though it is an estimate rather than an equality and you should use Google’s own count-tokens endpoint when the number has to be right. And it means the vocabulary was sized for a model family where a 256K embedding table is a rounding error, then inherited by models where it is not.

What it does not mean is that the two are interchangeable. A subset is not the whole, special tokens differ, and Gemini’s hosted API counts multimodal content in units that have nothing to do with a text tokenizer. Use the local tokenizer for capacity planning and cost estimation; use the API for anything you bill on or enforce a limit with.

The lineage is also why Gemma checkpoints ship a SentencePiece model file rather than a byte-pair-encoding merges table of the kind several other open families use. The practical difference is in the tooling: a SentencePiece vocabulary is a single binary artefact, so reimplementing the tokenizer outside Python means loading that file with a SentencePiece binding rather than porting a merge list. Any reimplementation should be checked against the reference by round-tripping a corpus and comparing ids, not by eyeballing a sentence.

Digits, bytes and whitespace

• Digits are split individually. The number 4711 becomes four tokens rather than one piece the model has to memorise. This is a deliberate choice across the Gemini and Gemma line and it measurably helps arithmetic, at the cost of making numeric text longer.
• Byte-level fallback. Anything the vocabulary does not cover falls back to raw bytes rather than to an unknown token, so there is no input the tokenizer cannot round-trip. Rare scripts and unusual symbols get expensive rather than lossy.
• Whitespace is preserved. Runs of spaces and indentation survive tokenization, which is what makes the tokenizer usable for code without mangling it.
• The turn markers are single tokens. <start_of_turn> and <end_of_turn> are vocabulary entries, which is why the chat template costs so little to apply: four extra tokens per turn, not forty characters.

What a huge vocabulary costs a small model

An embedding table is vocabulary size times hidden dimension, and it appears twice if input and output embeddings are untied. At 256K entries this dominates the small end of the family: a meaningful share of Gemma 2 2B’s parameters are embeddings rather than transformer layers, which is why its headline parameter count overstates how much computation happens per token relative to a model with a 32K vocabulary at the same size.

What it buys is fewer tokens per unit of meaning, especially outside English, and a softmax over a vocabulary fine-grained enough to make those tokens meaningful. On a model with an 8,192-token window, fewer tokens per sentence is not a cosmetic win: it is more conversation inside a fixed budget. The vocabulary choice and the short context window partially cancel each other out.

There is a second benefit at the output side that is easy to miss. A model produces one token per forward pass, so text that is ten per cent shorter in tokens takes ten per cent fewer passes to generate. Vocabulary size buys latency, not only context. And the extra parameters are not wasted capacity in the way the arithmetic suggests: embeddings are a lookup rather than a matrix multiply, so they cost memory and almost no compute per token. A 2B model with a 256K vocabulary does less arithmetic per token than its parameter count implies, which is part of why it runs well on modest hardware.

The cost lands somewhere else instead: the output projection. Every generated token requires scoring all 256,000 entries, and that softmax is a real matrix multiply over the full vocabulary at every step. On the smallest sizes it is a visible fraction of generation time. This is the trade the design makes explicit — cheaper prompts and slightly more expensive steps.

Why your token counts differ from other families

If you are comparing Gemma against another open family on cost or on context pressure, count with each family’s own tokenizer. The same paragraph does not produce the same number of tokens on a 256K vocabulary and a 128K one, and the gap widens on non-English text, long identifiers and numerals.

from transformers import AutoTokenizer

text = open("prompt.txt", encoding="utf-8").read()
for repo in ["google/gemma-2-9b-it", "google/gemma-3-4b-it"]:
    tok = AutoTokenizer.from_pretrained(repo)
    print(repo, len(tok(text)["input_ids"]))

Note the shape of the answer: it is per revision, not per family. Gemma 3 changed vocabulary, so a count taken against Gemma 2 is not transferable, and the same caution applies whenever you move a pinned checkpoint forward.

Two habits follow from that. Measure on your own corpus rather than on a sample paragraph, because the gap between tokenizers depends entirely on what you send: English prose separates them by a few per cent, while transliterated names, Indic and Southeast Asian scripts, and identifier-dense code can separate them by a third. And measure the rendered chat template, not the raw message content, since the per-turn formatting overhead differs between families as much as the text does.

One more caution about the headline number. A large vocabulary reduces the token count for text the vocabulary covers well, and Gemma’s was fitted on Google’s pretraining mixture rather than on yours. Domain jargon, product identifiers, chemical names and internal codes are unlikely to be single pieces in any general vocabulary, so a corpus full of them will not see the benefit the number suggests. If tokenization efficiency is load-bearing for your costs, the only meaningful measurement is tokens per character on your own text, and producing it takes about a minute.

Finally, do not use one family’s tokenizer as a proxy for another’s billing. A count from Gemma’s tokenizer is a good estimate for Gemma, a fair one for Gemini given the shared lineage, and a poor one for anything else. Providers bill on their own count, and the gap between your estimate and their invoice is a reconciliation problem you can avoid by counting with the right tokenizer from the start.