Why Kazakh Text Costs More Tokens Than English

Kazakh is a Turkic language written in an extended Cyrillic alphabet. Those two facts are independent, they both cost tokens, and they do not simply add up — the script penalty applies to every byte of every suffix the morphology produces, so the two compound.

Two costs, not one

The script cost is mechanical. Cyrillic occupies U+0400–U+04FF with a supplement above it, all of which is below U+0800, so every Kazakh letter is two bytes in UTF-8 rather than one. A Kazakh string is roughly twice its character count in bytes before any linguistics enter the picture.

The morphological cost is different in kind. Kazakh is agglutinative: grammatical relations that English expresses with separate words are expressed as suffixes glued to the stem in a fixed order — plural, then possessive, then case. A single Kazakh orthographic word routinely encodes what English spreads across four or five. That is fewer words, but each one is long, and long words are exactly what a subword vocabulary handles worst when it has not seen many of them.

These compound because the suffix chain is made of Cyrillic letters. Each suffix the morphology adds is not one extra character but two extra bytes per character, and the fragments a tokenizer splits it into are themselves two-byte-per-character fragments. The penalty is multiplicative in a way it is not for Azerbaijani, which has the same morphology in Latin script — that pair is the controlled comparison, and it is worth reading the two pages together.

Deriving the multiplier

Kazakh  : Кітапханада қазақ тіліндегі кітаптар көп.
English : There are many Kazakh-language books in the library.

Kazakh  : 36 Cyrillic letters      = 36 × 2 bytes = 72
          4 spaces + 1 full stop   =  5 × 1 byte  =  5
                                     total        = 77 bytes
          5 orthographic words

English : 52 ASCII characters      = 52 bytes
          9 orthographic words

English at roughly four characters per token gives about 13 tokens. Kazakh’s upper bound is 77 tokens — one per byte, the value you get under complete byte fallback. Its lower bound, if every one of the five words were a single learned token, is 5. The derived range is therefore 0.4x to 5.9x, which is uselessly wide until you narrow it with something known.

What is known is that a Cyrillic vocabulary trained on web text has plenty of Russian merges and very few Kazakh ones, so the realistic position is much closer to the byte-fallback end for Kazakh-specific material and much closer to the middle for the Russian loanwords and international vocabulary that Kazakh text also contains. A mixed document lands somewhere around three to five times English. This is derived reasoning about where in a bounded range the answer sits, not a measurement, and the section below tells you how to replace it with one.

The nine letters Russian does not have

Kazakh Cyrillic adds nine letters to the Russian inventory: ә ғ қ ң ө ұ ү һ і. They are charted by the Unicode Consortium in the Cyrillic block, and their distribution in training data is the interesting part.

• і (U+0456) is the dotted i. Its overwhelming corpus presence is Ukrainian, not Kazakh. Any merge involving it was almost certainly learned from Ukrainian word shapes, which do not resemble Kazakh ones.
• ә (U+04D9), қ (U+049B), ғ (U+0493) appear in Kazakh, Bashkir, Tatar and a handful of other Turkic languages of the region, all of them small in a web crawl. These are the letters most likely to sit at byte fallback.
• ө (U+04E9) and ү (U+04AF) also occur in Mongolian, which shares the extended Cyrillic convention. A merge learned from Mongolian will fire on Kazakh text, but on Mongolian syllable boundaries.

The practical effect is that a Kazakh word is not tokenized as a Kazakh word. It is tokenized as a sequence of fragments, several of which were learned from Ukrainian and Mongolian, and the boundaries therefore fall in linguistically meaningless places — which matters beyond cost, because it is also what degrades embedding quality on short queries.

What agglutination does to a merge table

Consider кітап (book) and what Kazakh builds from it: the plural, the possessive, the locative, the ablative, and combinations of all of them, each with a front or back vowel variant chosen by vowel harmony. One stem generates dozens of surface forms, and vowel harmony means the same grammatical suffix is spelled two or four different ways depending on the stem.

BPE learns merges by frequency. A language that spells one suffix one way concentrates that frequency; Kazakh splits it across harmony variants, so each variant is individually rarer and each is individually less likely to clear the threshold for a merge. A language with rich morphology needs a larger share of the vocabulary to reach the same efficiency as an isolating language, and it almost never gets one. This is the same effect described for tokenizers generally by Petrov and colleagues in “Language Model Tokenizers Introduce Unfairness Between Languages” (2023).

The consequence for a budget: because Kazakh packs more meaning into each word, a Kazakh sentence has fewer words than its English translation, and people reasonably expect that to help. It does not. The five words above expand into something in the region of twenty to forty tokens where nine English words become thirteen.

Measuring it, and the Latin question

This script separates the two costs the page has been arguing about. It reports bytes per token, which isolates how well the vocabulary covers the script, and tokens per word, which isolates how badly the morphology is being fragmented.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
text = open("kazakh.txt", encoding="utf-8").read()

words  = text.split()
tokens = enc.encode(text)
bytes_ = len(text.encode("utf-8"))

print("bytes/token :", round(bytes_ / len(tokens), 2))   # ~1.0 = byte fallback
print("tokens/word :", round(len(tokens) / len(words), 2))

A bytes-per-token near 1.0 means the vocabulary has essentially nothing for the script and you are paying per byte. A tokens-per-word above about six means the morphology is being shredded. The two numbers point at different fixes: the first is a reason to compare providers, the second is a reason to shorten prompts and to size retrieval chunks in tokens.