Why Hungarian Text Costs More Tokens Than English

Hungarian and Finnish are distantly related, both heavily agglutinative, and both expensive to tokenize. They get there by different routes. In Finnish the stem changes. In Hungarian the stem is comparatively stable and the suffixes multiply, which produces a similar bill from a mechanism you fix differently.

Every suffix has two or three spellings

Hungarian vowel harmony requires suffix vowels to agree with the vowels of the stem. This is not a phonetic detail invisible in writing; it is spelled out, so most suffixes exist in two or three written forms.

• The inessive is -ban after a back-vowel stem and -ben after a front-vowel stem.
• The instrumental is -val or -vel, before assimilation does further damage to it.
• Three-way alternations exist where rounding matters too: -hoz, -hez, -höz for the allative.

For a byte-level tokenizer this is a straightforward frequency division. The suffix that carries one grammatical function appears as two or three distinct byte sequences, each at a fraction of the combined frequency. Where an invariant suffix would have earned a merge slot comfortably, each variant now competes on its own reduced count, and the rarest of a three-way set may earn none.

The same division applies to the stem-plus-suffix combinations that BPE would otherwise have learned wholesale. A frequent noun in the inessive is not one string but one string per harmony class the noun belongs to, and the paradigm the vocabulary might have memorised is instead spread thin.

Assimilation dissolves the join

The instrumental suffix does something worse than alternate. Its initial v assimilates to a stem-final consonant and the result is written with a doubled consonant: kés, knife, plus -vel gives késsel. The v is gone from the spelling entirely.

A tokenizer looking for the suffix cannot find it, because in the written form there is no suffix-shaped byte sequence to find — only a doubled consonant that happens to be the surface of a morpheme boundary. The merge learned for the bare stem does not extend, and the merge learned for the suffix does not apply. What the tokenizer sees is an unfamiliar word.

Because assimilation depends on the stem’s final consonant, the instrumental of every noun in the language is a differently-shaped string. There is no shared fragment across the paradigm to anchor on. This is the single most expensive construction in Hungarian for a BPE tokenizer, and it is not present in Finnish at all.

Two vowels that cost more than the others

Hungarian marks vowel length and quality with acutes and double acutes: á, é, í, ó, ő, ú, ű. All are two bytes in UTF-8, so on the script axis Hungarian pays a small, uniform premium over English, comparable to German’s.

But two of those characters are not like the others. The vowels á, é, í, ó and ú live in the Latin-1 Supplement and appear across French, Spanish, Portuguese, Italian, Icelandic and Czech, so they occur constantly in training corpora and the byte sequences around them are heavily merged. The double-acute vowels ő (U+0151) and ű (U+0171) live in Latin Extended-A and are used, in ordinary running text, essentially only by Hungarian.

Both are still two bytes — the script axis does not distinguish them. The difference is on the merge axis: a byte pair that occurs only in Hungarian text competes for a vocabulary slot with only Hungarian’s corpus share behind it, while á rides on the combined frequency of a dozen languages. So within a single Hungarian word, some accented vowels tokenize well and some reliably force a split. Words containing ő and ű are systematically more expensive than their vowel-harmony counterparts containing ó and ú, which is a genuinely odd property and a real one.

Deriving the multiplier

Assumptions: English at four characters per token, the rule of thumb OpenAI publishes for counting tokens. Hungarian at about 1.08 bytes per character — higher than Finnish’s because accented vowels are denser. Hungarian merge efficiency of 2.2 to 2.8 bytes per token, on the harmony and assimilation arguments above. Length factor of 1.05 characters per English character: Hungarian’s suffixes absorb prepositions, articles and possessives that English writes separately, so its words are long but its sentences are not proportionally longer.

At m of 2.5, Hungarian is 1.08 divided by 2.5, or 0.432 tokens per character, against English’s 0.25 — a per-character ratio of about 1.73x. With the length factor, a per-meaning ratio near 1.81x. A 200-character English sentence at 50 tokens is roughly 91 tokens of Hungarian.

That lands close to the Finnish figure derived on the neighbouring page, from different inputs. The coincidence is real and worth understanding rather than smoothing over: two unrelated mechanisms, stem mutation and suffix allomorphy, both have the effect of splitting a paradigm’s frequency across byte sequences that share no merge, and the size of that effect is set by the number of resulting variants rather than by which part of the word varies.

Why this is not the Finnish page

The reason to keep them separate is that the fixes differ.

• For Finnish, lemmatisation collapses gradation alternants onto one stem and buys a large recall improvement in retrieval, because the alternants are genuinely different strings for the same word.
• For Hungarian, lemmatisation helps less on the stem — which was mostly stable anyway — and more on the suffix, and it must handle assimilated forms, which requires a real morphological analyser rather than a suffix-stripping heuristic. A stripper that removes -vel will never fire on késsel.
• For both, the token budget correction is similar in size, so if all you are doing is sizing context windows you can treat them alike. If you are building retrieval, you cannot.

import tiktoken, unicodedata
enc = tiktoken.get_encoding("o200k_base")

def t(s):
    return len(enc.encode(" " + unicodedata.normalize("NFC", s)))

# does the double acute cost extra? compare harmony pairs
for a, b in [("bokor", "tükör"), ("hosszu", "hosszú"), ("ero", "erő")]:
    print(f"{a:10s} {t(a)}    {b:10s} {t(b)}")

# does assimilation break the suffix merge?
for base, inst in [("kés", "késsel"), ("hajó", "hajóval"), ("víz", "vízzel")]:
    print(f"{base:8s} {t(base)}  ->  {inst:10s} {t(inst)}")

If the instrumental forms cost two or more tokens above their bases while the non-assimilating ones cost one, you have observed the mechanism directly. For the broader family of assumptions that break on languages shaped like this, see agglutinative languages and NLP assumptions.