Why Tokenizer Vocabulary Size Is a Bottleneck for Low-Resource Languages

A tokenizer vocabulary is a fixed number of slots, and those slots are handed out in proportion to how often a pattern appears in the training corpus. That single design choice, applied to a corpus that is overwhelmingly English, is most of the reason a sentence in Amharic costs several times what the same sentence costs in English.

The vocabulary is a fixed budget

Modern models use subword tokenizers — byte-pair encoding or a unigram variant — with a vocabulary fixed before training begins. Typical sizes run from about 32,000 entries for older models to roughly 100,000–256,000 for recent ones. That number cannot grow afterwards. It is baked into the embedding matrix and the output layer, and both scale linearly with it: a vocabulary of 256,000 with a hidden size of 8,192 costs about two billion parameters in the embedding table alone, plus the same again if the output projection is not tied to it.

So vocabulary size is a real budget with a real price, and the price is paid by every user of the model on every token, not just by the speakers of the languages that benefit. That is why nobody simply makes it ten times larger, and it is the constraint every argument on this page runs into.

How merges get allocated

BPE training is a greedy loop. Start with the base alphabet — in modern byte-level BPE, the 256 possible byte values. Then repeatedly find the most frequent adjacent pair in the corpus, add it to the vocabulary as a new token, and replace every occurrence. Stop when the vocabulary reaches its target size.

The important word is frequent. Every merge is spent on whatever is most common in the corpus as a whole. There is no per-language quota, no fairness constraint, and no notion that a language exists. If English is 45% of the corpus, English patterns win the early merges, and once the common English words are single tokens the loop moves on to common English word pieces, then to common English word pieces with leading spaces, then to punctuation patterns in English code and markup — all of which are still more frequent than a common word in a language holding 0.001% of the corpus.

The arithmetic for a rare language

Take a concrete, clearly-labelled hypothetical. Assume a 100,000-entry vocabulary, 256 of which are byte values, leaving 99,744 merges to allocate. Assume a language holds 0.001% of the corpus by character count — not an unreasonable share for a language with a few million speakers and a thin web presence. If merges were allocated in strict proportion to corpus share, that language’s allocation is:

merges available        = 100,000 - 256      = 99,744
language's corpus share = 0.001%             = 0.00001
proportional allocation = 99,744 x 0.00001   = 0.997 merges

  ~1 merge for the entire language.

One merge. In practice the allocation is worse than proportional at the low end and better at the high end, because BPE is greedy rather than proportional: it takes the single most frequent pair each time, which systematically favours whatever is already dominant. And it is better than this in one respect — a language sharing a script with a well-represented one inherits merges it did not pay for. Swahili written in Latin script gets English and Spanish letter-pair merges for free; Amharic in Geʽez script inherits nothing, because no well-represented language uses those code points.

Now run the same arithmetic for English at 45% of the corpus: 99,744 × 0.45 ≈ 44,885 merges. Enough for tens of thousands of whole words. The ratio between the two allocations is about 45,000 to 1, and that ratio — not any decision about which languages matter — is what produces the cost differences catalogued across the per-language token cost pages.

What byte fallback actually costs

A language that wins no merges does not fail to tokenize. Byte-level BPE always has the 256 byte values as a floor, so any text encodes — one token per UTF-8 byte in the worst case. That is the mechanism people mean by “byte fallback”, and its cost is set by UTF-8 itself:

• ASCII — 1 byte per character. Worst case, 1 token per character.
• Latin with diacritics, Greek, Cyrillic, Hebrew, Arabic — 2 bytes per character. Worst case, 2 tokens per character.
• Devanagari, Geʽez, Thai, Khmer, CJK — 3 bytes per character. Worst case, 3 tokens per character.
• Emoji and less common CJK extensions — 4 bytes. Worst case, 4 tokens per character.

So the floor for an unmerged Geʽez or Devanagari text is roughly three tokens per character, against roughly one token per four characters for well-tokenized English. That is a ratio around 12:1 on the same semantic content, before any consideration of morphology. Add an agglutinative or heavily inflected grammar on top — where a single word carries what English spreads across five — and the two effects partly cancel, which is why measured fertility ratios usually land between 2:1 and 8:1 rather than at the theoretical extreme.

Three consequences beyond price

The bill is the visible consequence and the least interesting one.

Effective context shrinks. A 128,000-token context window is a document-length limit only if your tokens are dense. At 3 tokens per character you fit roughly 43,000 characters, against roughly 500,000 for well-tokenized English. The same model advertises the same window and delivers an order of magnitude less document.

Learning is harder, not just costlier. A word split into eleven byte fragments gives the model eleven positions across which to learn one unit of meaning, and the fragments are shared with every other language using the same bytes. Signal that would be concentrated in one embedding is smeared across a shared, ambiguous set. This is a plausible part of why low-resource languages produce more fabrication, alongside the more obvious explanation that there was less to learn from.

Chunking heuristics break. Any pipeline that chunks by character count and assumes a token ratio will overflow silently for these languages, which is the failure behind sizing chunks in tokens rather than characters.

What can be done about it

Little, at the point where you are calling an API. The tokenizer shipped with the model and cannot be swapped. The realistic levers are to measure fertility for your actual text against each candidate model before committing — tokenizer behaviour on non-Latin scripts varies far more between model families than their pricing pages suggest — and to budget in tokens rather than characters from the start.

Upstream, the fixes are structural: vocabulary allocation that reserves a floor per script rather than following raw frequency, corpus upsampling for under-represented languages during tokenizer training, and purpose-built tokenizers for a specific language. All three exist in published work; none of them help the model you are calling today.