Why Icelandic Text Costs More Tokens Than English

Icelandic has none of the excuses. It is written left to right in Latin letters, it is a European language with full national digital infrastructure, its orthography is regular and its texts are online. It tokenizes badly anyway, and the reason is arithmetic about corpus size that applies to every small language regardless of script.

The merge budget is a frequency auction

Byte pair encoding, as described in Sennrich, Haddow and Birch’s original 2015 paper, works by repeatedly finding the most frequent adjacent pair of symbols in the training corpus and merging it into a new symbol. Repeat until the vocabulary reaches its target size.

Sennrich, Haddow and Birch, “Neural Machine Translation of Rare Words with Subword Units” (2015) is the source of the algorithm every production tokenizer still uses, and reading its procedure makes the consequence obvious. Merges are awarded strictly by global frequency. There is no per-language quota, no floor, no fairness constraint. A byte sequence that is enormously frequent within Icelandic but rare in the corpus as a whole loses every round to a sequence that is moderately frequent in English.

So a language’s tokenization quality is essentially a function of its share of the training corpus, and the relationship is worse than linear: because merges compound — a merge for a frequent fragment enables longer merges built on it — a language above the threshold gets compounding returns and one below it gets nothing.

Icelandic’s share of the auction

Do the arithmetic with round numbers and it is stark. Common Crawl publishes a language breakdown of its crawls; English is the plurality by a very wide margin and the distribution falls off steeply into a long tail. Icelandic sits deep in that tail, with a speaker population on the order of a few hundred thousand — Statistics Iceland publishes the current national figure, which has been under half a million throughout the modern web era.

Common Crawl publishes per-language page counts for each crawl, which is the right place to check the share for whichever crawl a model you care about was trained on, rather than taking a number from here.

Suppose Icelandic is one part in twenty thousand of a corpus. A 200,000-entry vocabulary allocating slots in proportion would give it ten. Ten merges is not a language; it is a handful of the most common function words. In practice Icelandic does better than that, but not because it won the auction — because it shares byte sequences with English, Danish, Norwegian and German, and inherits their merges wherever the spelling happens to coincide.

Two letters with no neighbours

That inheritance is exactly where Icelandic’s specific problem sits. The Icelandic alphabet contains þ (thorn, U+00FE) and ð (eth, U+00F0), both two bytes in UTF-8. Both are used in ordinary running text by essentially no other living language — Faroese uses eth, and that is a population smaller again.

The accented vowels á, é, í, ó, ú and ý are shared with Spanish, Czech, Hungarian, Portuguese and others, so the byte pairs involving them have been merged on the strength of those languages’ corpus shares. Icelandic gets those merges for free. It gets nothing for free around thorn and eth.

And thorn and eth are not marginal. They spell the most frequent words in the language: the definite forms, the demonstratives, the second person pronoun, and the past tense endings of a large class of verbs. So the characters Icelandic cannot borrow merges for are the ones it uses most, and every occurrence of them tends to force a token boundary. A word like það, three characters, is four bytes and quite plausibly two or three tokens where an English word of comparable frequency is one.

The same argument, in a milder form, applies to the Nordic vowels å, ä and ö across the mainland Scandinavian languages; the page on Nordic characters in AI output covers what happens when a model has to generate them rather than read them.

A vocabulary deliberately unshared with English

One more Icelandic-specific factor, and it is a matter of policy rather than orthography. Icelandic has a long-standing tradition of linguistic purism: rather than borrowing international technical vocabulary, new terms are coined from native roots. Computer is tölva, telephone is sími, and the pattern extends through most of the modern technical lexicon.

Most European languages get a substantial discount on technical text because their technical vocabulary is a lightly respelled version of the English one, and the English merges nearly fire. Icelandic gets none of that discount. Its technical vocabulary shares no bytes with the corpus that funded the merge table. So the domain where you would most expect a European language to tokenize well — software documentation, technical support, product copy — is the domain where Icelandic does worst.

Deriving the multiplier

Assumptions: English at four characters per token, per OpenAI’s published rule of thumb. Icelandic at about 1.1 bytes per character, allowing for a fairly high density of thorn, eth and accented vowels. Icelandic merge efficiency of 1.8 to 2.4 bytes per token, on the corpus-share argument — notably worse than German or Finnish, because the corpus share is smaller by orders of magnitude and the unshared characters are the frequent ones. Length factor of 1.1 characters per English character: Icelandic inflects for four cases and three genders and writes compounds closed, so its words run long.

At m of 2.1, Icelandic is 1.1 divided by 2.1, or 0.524 tokens per character, against English’s 0.25 — a per-character ratio of about 2.1x. With the length factor, a per-meaning ratio near 2.3x. A 200-character English sentence at 50 tokens becomes roughly 115 tokens of Icelandic.

That is a smaller multiplier than any three-byte script in this cluster and a larger one than any other Latin-script language in it, which is exactly what the two-axis model predicts: Icelandic loses badly on the merge axis and barely at all on the script axis.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

# how much does a single thorn or eth cost?
probes = ["that", "the", "there", "það", "þá", "þetta", "sími", "tölva"]
for w in probes:
    n = len(enc.encode(" " + w))
    b = len((" " + w).encode("utf-8"))
    print(f"{w:8s} {b:2d} bytes  {n:2d} tokens  m={b/n:4.2f}")

The m column separates the two axes cleanly. If the Icelandic entries come back near 1.0 while the English ones are near 4.0, you are looking at the merge auction rather than at UTF-8, and the general form of that problem is covered in the tokenizer vocabulary bottleneck for low-resource languages.