Why Turkish Text Costs More Tokens Than English

Turkish is the language in this cluster where the byte argument fails completely. A Turkish sentence is roughly the same size in UTF-8 as its English translation, and it still tokenises worse. Everything interesting about Turkish token cost is in the morphology.

Almost the same bytes, more tokens

Bu işlev çok fazla bellek kullanır.
"This function uses a lot of memory."

Count the Turkish: 34 code points, of which four are non-ASCII — ş (U+015F), ç (U+00E7) and ı (U+0131), all below U+0800 and therefore two bytes each. That is 30 single-byte characters plus 4 two-byte ones: exactly 38 bytes. The English is 35 bytes. A byte multiplier of 1.09, which is nothing.

Derive the tokens with the assumption stated. Assume the tokenizer holds merges for common Turkish stems and for the highest-frequency suffixes, but that most inflected forms break into a stem piece plus one or two suffix pieces, and that the non-ASCII letters terminate merges wherever they appear. Six words of that shape gives roughly 12 to 20 tokens against the English 9 — a derived multiplier of about 1.4× to 2.2×, against a byte multiplier of 1.09.

That gap is the entire subject of this page. Turkish is not expensive because of Unicode. It is expensive because of what a merge table can afford to remember.

One word, eight English words

Turkish is agglutinative: grammatical relations are expressed by appending suffixes to a stem, each suffix contributing one clean unit of meaning, and the chain can be long. The classic teaching example, and it is a real word rather than a constructed one, is kullanabileceklerimizden — from kullan (use) plus abil (ability) plus ecek (future) plus ler (plural) plus imiz (our) plus den (ablative). Roughly: “from those that we will be able to use”. One Turkish word, eight English ones.

For a byte-level BPE tokenizer this is close to the worst possible input shape. The word is 24 characters and about 26 bytes; it is a single pre-token, so all the merging must happen inside it; and the full string will essentially never have appeared often enough in a training corpus to earn merges of its own, because the number of possible suffix chains is combinatorial. The tokenizer falls back to whatever stem and suffix fragments it does know, which is typically four to eight pieces.

The compensation is that Turkish uses far fewer words. Comparing word counts between Turkish and English is meaningless in the same way comparing character counts between Chinese and English is meaningless, and for the mirror-image reason: Turkish packs more into a word, Chinese packs more into a character, and only the token count is comparable. The same argument applies to Finnish and Hungarian, which are agglutinative for the same structural reasons and are not re-derived here.

Vowel harmony multiplies the merge table

Here is the Turkish-specific mechanism that makes it worse than agglutination alone would. Turkish suffixes obey vowel harmony: the vowels in a suffix change to match the vowels in the stem. The plural suffix is -ler or -lar. The locative is -de, -da, -te or -ta. The possessive third person is -ı, -i, -u or -ü. The copula is -dır, -dir, -dur, -dür, -tır, -tir, -tur or -tür.

So a single grammatical morpheme occupies two, four or eight distinct byte sequences depending on its phonological environment. A merge table that wants to represent Turkish plurals as one token has to spend two slots, not one; the copula costs it eight. Multiply that across the whole suffix inventory and Turkish demands several times as much vocabulary as its morpheme count suggests, competing for slots against languages that need one sequence per morpheme.

Consonant mutation compounds it at the other end. Stem-final p, ç, t and k voice to b, c, d and ğ before a vowel-initial suffix: kitap becomes kitabı, not kitapı. So the stem itself changes spelling when suffixed, which means a merge learned for the bare stem does not fire on the inflected form. Both ends of the word move.

Six letters that break merges mid-word

Turkish adds six letters to the Latin alphabet: ç, ğ, ı, İ, ö, ş and ü. Every one is non-ASCII and two bytes. Unlike French or German accents, which cluster in predictable places, these appear throughout ordinary Turkish words — ı in particular is one of the eight Turkish vowels and is extremely common.

The effect is that a Turkish word is frequently an ASCII fragment, a two-byte letter, another ASCII fragment. The merge table’s English investment covers the fragments and stops at each non-ASCII letter, so words are chopped at their vowels. This is why the token multiplier exceeds the byte multiplier by so much: the non-ASCII letters cost little in bytes and a great deal in merge continuity.

The dotted and dotless i pair deserves separate mention because it causes bugs beyond token counting. Turkish has four i-letters: ı and I as one pair, i and İ as another. Uppercasing i in the Turkish locale gives İ, not I, and lowercasing I gives ı, not i. A locale-unaware toLowerCase() in a normalisation pipeline will therefore silently corrupt Turkish text and change its token count — see the dotted and dotless i problem.

Measuring it, and what it changes

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

chain = ["kullan", "kullanabil", "kullanabilecek",
         "kullanabilecekler", "kullanabileceklerimiz",
         "kullanabileceklerimizden"]

for w in chain:
    s = " " + w
    print(w,
          "chars", len(w),
          "bytes", len(s.encode("utf-8")),
          "tokens", len(enc.encode(s)))

Read the token column as the suffix chain grows. Each added suffix should add roughly one token if the tokenizer knows that suffix, and two or more if it does not. That per-suffix cost is the number to use when estimating Turkish output length, and it is far more useful than a whole-document ratio because it tells you which morphemes your tokenizer actually covers.

• Do not size anything in words. “Answer in 50 words” means something very different in Turkish than in English, and the model will comply with the instruction rather than with your intent.
• Budget output tokens generously. A Turkish answer of the same content is more tokens, so a max_tokens value tuned on English truncates it mid-word — and mid-word in Turkish means mid-suffix, which changes the grammatical meaning of what survives.
• Stemming helps keyword search and hurts embeddings. The suffixes carry case, number, person and tense; stripping them for BM25 is standard, stripping them before embedding discards real signal.