Tokenizer Differences Between Model Families

Two models quoted at the same price per million tokens can differ by a third in what they cost to run on identical text, because they disagree about how many tokens that text is. The disagreement is systematic, and it is a property you can look up rather than guess.

The families, and what they inherit

Almost every production tokenizer descends from one of two lineages, and knowing which one tells you most of its behaviour:

The lineage also predicts the annoyances. Byte-level BPE will encode anything but treats a Japanese sentence as a run of byte fragments unless the merge table covers it. SentencePiece normalises whitespace in ways that occasionally surprise you when round-tripping code.

What vocabulary size buys

The single most predictive published number is the vocabulary size, and the trend across generations is unmistakable. These are the figures in each tokenizer’s own configuration — check the config you are actually loading, since forks and fine-tunes add tokens:

Bigger vocabularies compress text into fewer tokens, which means cheaper requests, more content per context window, and fewer decode steps per sentence. The cost is paid in the model itself: the embedding matrix and the output projection both scale with vocabulary size, and for a small model that overhead is a meaningful fraction of the total parameter count. That is the whole trade, and it is why small models keep small vocabularies while frontier models can afford 200k+.

One caveat on reading these numbers: a fine-tune can add tokens to a published vocabulary, so the config in a derived repository may not match the base model’s. If you are counting for a fine-tune, load that repository’s tokenizer rather than its parent’s. Special tokens for tool calling and for reasoning delimiters are the usual additions, and they are exactly the ones that appear in every request.

Prices are not comparable until you normalise

This is the practical consequence and it is routinely missed. Suppose you are choosing between two models on your own corpus:

                 sticker price     your corpus      cost per 1M chars
  Model A        $1.00 / M tok     3.5 chars/tok    1.00 / 3.5 = $0.286
  Model B        $0.90 / M tok     4.6 chars/tok    0.90 / 4.6 = $0.196

  sticker difference:            B is 10% cheaper
  actual difference on your text: B is 31% cheaper

The chars-per-token figures are the ones you must supply — they are a joint property of the tokenizer and your text, and they are exactly what a generic comparison table cannot know. The arithmetic itself is trivial: price / chars_per_token is a cost per character, and cost per character is the only figure two model families can be compared on.

The same normalisation applies to context windows. A 128k window at 3.5 chars per token holds about 448,000 characters; at 4.6 it holds about 589,000. Comparing the windows without it overstates the smaller-vocab model by about a third.

And it applies to the output side twice over, which is the part almost nobody adjusts for. A model with a coarser tokenizer needs fewer tokens to write the same answer, so it is cheaper per answer at equal per-token output pricing — and it is also faster per answer at equal tokens per second, because it performs fewer decode steps to say the same thing. Which means a published throughput figure in tokens per second is not comparable across families at all. Two models at 80 tokens per second, one emitting 3.5 characters per token and the other 4.6, differ by 31% in the only rate a user perceives, which is characters on screen per second. If you are benchmarking latency across families, normalise to characters or you are measuring the tokenizer.

Building the table on your own text

from transformers import AutoTokenizer

REPOS = ["<repo-a>", "<repo-b>", "<repo-c>"]     # the models you are choosing between
PRICES = {"<repo-a>": 1.00, "<repo-b>": 0.90}    # $ per 1M input tokens

text = open("representative_sample.txt", encoding="utf-8").read()
print(f"{'tokenizer':28s} {'vocab':>8s} {'tokens':>9s} {'ch/tok':>7s} {'$/Mchar':>9s}")

for repo in REPOS:
    tok = AutoTokenizer.from_pretrained(repo)
    n = len(tok.encode(text, add_special_tokens=False))
    cpt = len(text) / n
    cost = PRICES.get(repo, float("nan")) / cpt
    print(f"{repo:28s} {tok.vocab_size:8d} {n:9d} {cpt:7.2f} {cost:9.3f}")

Two rules for the sample file. It must be your real content — the mix of languages, code, markup and boilerplate you actually send — because any other text measures a corpus you do not have. And run it separately per language and per document type, since the aggregate hides exactly the cases where the families diverge most, which is the subject of the language tax page.

Add the output side while you are at it, since the script above only measures input. Run the same fifty representative prompts through both candidates and count the tokens in the answers they produce. A model that is more verbose, or whose tokenizer is coarser, will differ from the other on the expensive half of the bill in a way no price sheet can tell you. It is an hour of work and it is the only part of a model-cost comparison that cannot be looked up.