Why Finnish Text Costs More Tokens Than English

Finnish is written in plain Latin letters, is a well-documented European language with a substantial digital corpus, and still tokenizes badly. The usual explanation — that it is agglutinative — is only half of it. Agglutination alone is survivable. What is not survivable is that the stem changes shape underneath the suffixes.

How many forms one Finnish noun has

Start with the arithmetic that makes vocabulary coverage hopeless. Finnish grammars count fifteen cases. Each combines with singular and plural. Each of those combines with a possessive suffix, of which there are six including the third-person form. So the core paradigm for a single noun is on the order of fifteen times two times seven, counting the no-possessive option — around 210 written forms before anything else is added.

BPE has a fixed vocabulary. o200k_base holds roughly 200,000 entries, shared across every language, every programming language and every piece of markup in the training corpus. If Finnish were allocated a generous one per cent of that, two thousand slots, it could hold complete paradigms for about ten nouns. In practice the slots go to the highest frequency forms across the whole language — the nominative singular of common nouns, the most frequent case endings as standalone fragments — and everything else is assembled.

Assembly is not a disaster in itself. A word split into a stem token and two suffix tokens is three tokens where English needed one or two. That is a ratio near 2x, which is roughly where a purely agglutinative language with good corpus coverage would land.

Gradation breaks the stem’s own merge

Here is the Finnish-specific mechanism. Consonant gradation alternates the stem’s medial consonants depending on whether the following syllable is open or closed. The textbook pairs are kk to k, pp to p, tt to t, k to nothing, p to v, t to d, and several more with nasal and liquid contexts.

The consequence for a byte-level tokenizer is precise and severe. The word katu, street, has a genitive kadun. Those two strings share the bytes for k and a and then diverge. The merge the vocabulary learned for katu, which is a frequent nominative and certainly present, does not fire on kadun at all. The inflected form has to be found from scratch, and the fragment kad is far less frequent than kat, so it is likelier to have no merge of its own.

Contrast that with a language where suffixes attach to an invariant stem. There the stem’s merge fires on every form in the paradigm, and the cost of inflection is one extra token for the suffix. In Finnish the paradigm is partitioned into a strong-grade set and a weak-grade set that share no useful merge, so the frequency that would have concentrated on one stem is split, and both halves land closer to the byte floor.

This is the difference between Finnish and Turkish, which is at least as agglutinative and has vowel harmony but no comparable stem mutation, and it is the difference from Hungarian, which reaches a similar ratio through allomorphy on the suffix rather than mutation of the stem.

The chain does not stop at the case ending

Finnish also permits enclitic particles after the possessive suffix — -kin for “also”, -kaan in negative contexts, -han, -pa, the interrogative -ko. These stack. The standard teaching example builds taloissammekin from talo, house: plural, inessive case, first person plural possessive, and the additive clitic, all in one whitespace-delimited word meaning “also in our houses”.

English needs four separate words for that, all of them among the most frequent words in the language and therefore all single tokens. Finnish needs one word of fourteen characters that essentially no vocabulary contains. This is the shape of the whole problem: Finnish concentrates into one rare string the information English spreads across several extremely common ones, and BPE rewards commonness.

Deriving the multiplier

Assumptions, stated: English at four characters per token, the rule of thumb OpenAI publishes for counting tokens. Finnish at 1.03 bytes per character. Finnish merge efficiency of 2.2 to 2.8 bytes per token, on the gradation argument above — worse than German’s despite a comparable script, better than any three-byte script. And a length factor: Finnish writes the same content in about 1.1 characters per English character, because the compression of function words into suffixes partly offsets the length of the resulting words.

At m of 2.5, Finnish is 1.03 divided by 2.5, or 0.412 tokens per character, against 0.25 — a per-character ratio of about 1.65x. With the 1.1 length factor the per-meaning ratio is about 1.81x. A 200-character English sentence at 50 tokens becomes roughly 91 tokens in Finnish.

As with German, the average hides the tail, and in Finnish the tail is driven by long inflected forms of low-frequency stems: technical vocabulary, place names in oblique cases, and compounds, which Finnish also writes closed. A Finnish address in the adessive case is close to a worst case for a tokenizer.

What to do about it

1. Measure your own ratio before you plan around anyone else’s. Encode a few hundred aligned Finnish and English sentences and divide.
2. Set context and chunk budgets in characters of source text, converted through your measured ratio, rather than reusing a token number derived from English documents.
3. For retrieval, consider lemmatising the indexed text. Finnish has mature open morphological analysers, and reducing gradation-alternating forms to a single lemma both improves recall and reduces the token count of the index.
4. Do not lemmatise the text you send for generation. The case endings carry the grammatical relations, and stripping them changes the meaning.
5. If output length matters, remember the ratio applies to completions too. A max-tokens limit tuned on English truncates Finnish answers mid-word, and mid-word truncation in a language with case endings produces text that is not merely cut short but ungrammatical.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

# does gradation actually cost you? compare strong and weak grade stems
for base, infl in [("katu", "kadun"), ("kauppa", "kaupan"), ("tyttö", "tytön")]:
    b = enc.encode(" " + base)
    i = enc.encode(" " + infl)
    print(f"{base:10s} {len(b)} tok   {infl:10s} {len(i)} tok")

If the inflected form costs more than one token above the base form, the stem’s merge is not surviving gradation, which is the effect this page is about and the reason Finnish sits where it does.