Token cost by language and script

A tokenizer is a lookup table built once, before training, from a corpus somebody chose. Modern ones are byte-level BPE: the text is encoded to UTF-8, every one of the 256 possible bytes is already a token, and on top of that sits a merge table — a learned list of byte pairs that get collapsed into single tokens, ordered by how often the pair appeared in that corpus. English is cheap not because English is simple but because the merge table is mostly English, so whole common words survive as one token. Everything below that is a fallback, and the last fallback is one token per byte. That floor is the single most useful fact in this cluster, because it means a script whose characters take three UTF-8 bytes cannot cost less than three tokens per character unless the merge table happens to contain that character.

So “language X costs more tokens” is never one reason. It is some combination of how wide the characters are in UTF-8, whether the orthography inserts combining marks and joiners that break merges apart, how much of the vocabulary budget the language was given, and how much meaning the language packs into a character in the first place. These pages take one language each and work out which of those is doing the damage, because the fixes differ: a vocabulary problem is solved by changing model, a byte-width problem is not solved at all, and a normalisation problem is solved before the text ever reaches the API. Published cross-language measurements exist — Petrov et al., “Language Model Tokenizers Introduce Unfairness Between Languages” (NeurIPS 2023) compares tokenised lengths of parallel text and reports ratios exceeding an order of magnitude between the best- and worst-served languages — and every page here tells you how to reproduce the number for your own corpus rather than quoting one.