Why Tokenizers Charge Non-English Users More
5 min read · updated August 3, 2026
Translate a paragraph into twelve languages and send each one to the same model. The meaning is constant, the price is not. The gap is not a pricing decision anybody made — it is an artefact of how the tokenizer was trained, and it is worth understanding because it is one of the few cost problems you can fix by changing model family.
Where the penalty comes from
A BPE tokenizer has a fixed budget: a vocabulary of some size, filled by merging the pairs that occurred most often in the tokenizer’s training corpus. If that corpus is 90% English, roughly 90% of the merge budget is spent on English strings. Common English words end up as one token each. A Hungarian or Telugu word that never crossed the frequency threshold gets no entry at all and is assembled from fragments.
For non-Latin scripts a second multiplier stacks on top. Byte-level BPE starts from the 256 UTF-8 bytes, and a Devanagari, Thai, Korean or Chinese codepoint occupies three bytes. If the merge table contains no entry covering that codepoint, one character costs three tokens before any word-level structure is considered. The worst case for a poorly covered script is therefore not “a bit more than English”; it is several tokens per character.
Worth being precise about the direction of causation, because it is often described as bias in the model and it is not. Nothing in the transformer prefers English. The penalty is entirely upstream, in a frequency table built once from a corpus, and it is therefore fixable by a vendor in a way that most fairness problems in machine learning are not: enlarge the vocabulary, rebalance the tokenizer corpus, and the tax falls without touching the model architecture at all.
What the research found
The number to quote here should come from a paper, not from a vendor blog. Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023, tokenized parallel corpora — the same content, professionally translated — across a large set of languages and compared the resulting sequence lengths. They reported that for some tokenizers the same content differs by more than an order of magnitude between the best- and worst-served languages, and framed the consequence exactly as a fairness problem: the speakers of those languages pay more per unit of meaning, wait longer, and fit less into the same context window.
That framing is the useful one. The tax is not primarily a translation quality problem. It is a unit-of-account problem, and it lands on the three bills below simultaneously.
It is three penalties, not one
| Penalty | Description |
|---|---|
| money | Directly proportional. If your Japanese documents tokenize at 2.5× the rate of the English ones, the input line of the invoice for those users is 2.5× larger for identical work. |
| capacity | A 128k-token window is not 128k tokens of content, it is 128k tokens of encoding. The same window holds a substantially shorter document in a poorly-served language, so a retrieval pipeline tuned on English can start silently truncating when it meets other locales. |
| latency | Output tokens are generated one at a time. If the answer needs more tokens to say the same thing, it takes proportionally longer to say it, regardless of how fast the hardware is. |
The direction of travel
This is improving, and the improvement is visible in a durable structural fact rather than a benchmark: vocabulary sizes have grown by roughly an order of magnitude across successive generations. GPT-2 shipped with a vocabulary of 50,257 entries; the cl100k_base encoding used by the GPT-4 generation has 100,277; Llama 2 used a 32,000-entry SentencePiece vocabulary and Llama 3 replaced it with a 128,256-entry one, with the model card explicitly citing improved multilingual encoding efficiency as the reason. Gemma’s tokenizer is larger still at 256,000.
A bigger vocabulary is not free — the embedding matrix and the output projection both scale with it, which is why small models keep small vocabularies — but it is the lever that reduces the tax, and vendors have been pulling it. When you are choosing a family for a multilingual product, vocabulary size and its multilingual composition are a more predictive spec than the headline benchmark score.
What you can actually do
- Measure per locale, not in aggregate. Compute characters-per-token separately for each language you serve. An average over a mixed corpus hides exactly the population that is being overcharged.
- Set limits in tokens, never in characters. A 2,000-character cap on user input is a different amount of budget in every language. Tokenize and cap on the real count.
- Choose the family on the tokenizer, sometimes. For a workload dominated by one non-English language, a family whose vocabulary covers that script well can be cheaper per unit of meaning even at a higher per-token price. This is the same normalisation argument as on the tokenizer comparison page.
- Compress the structural part, not the language part. Boilerplate, markup and repeated scaffolding are language-neutral waste and are usually the largest single win available before you start trading away meaning.
- Do not silently pivot through English. Translating input to English, reasoning, and translating back does reduce token count, and it also loses register, idiom and named entities. It is a legitimate option for internal classification, and a bad one for anything a user reads.
One reporting habit is worth adopting alongside all of that: break your inference spend down by locale. It is a single dimension on an existing metric and it turns an invisible cross-subsidy into a number somebody owns. Teams that have it discover fairly quickly that a market they thought was marginal on revenue is disproportionate on cost, which is a pricing conversation rather than an engineering one — but only if the number exists.