Why Azerbaijani Text Costs More Tokens Than English

Azerbaijani and Kazakh are both Turkic, both agglutinative, both with vowel harmony and stacked case suffixes. Azerbaijani is written in a Latin alphabet and Kazakh in Cyrillic. That makes the pair close to a controlled experiment: whatever difference remains between their token multipliers is the price of the script.

The comparison this page exists to make

Most explanations of why a language costs more tokens collapse two separate causes into one sentence. A script cost comes from UTF-8: any character above U+007F takes at least two bytes, and a byte-level tokenizer with no merges for those sequences pays per byte. A morphological cost comes from word shape: a language that builds long words out of many morphemes has a large surface vocabulary, which a fixed-size merge table cannot cover.

Azerbaijani separates them because it is almost pure ASCII. Its alphabet adds only ç ə ğ ı ö ş ü to the Latin basics, and in ordinary prose those appear often but not on most characters. So Azerbaijani text is close to one byte per character, and any multiplier it still carries against English is morphology and training share rather than encoding.

Deriving the multiplier

Azerbaijani : Kitabxanada Azərbaycan dilində çoxlu kitab var.
English     : There are many Azerbaijani-language books in the library.

Azerbaijani : 47 characters, of which 4 are non-ASCII (ə ə ə ç)
              43 ASCII characters      = 43 × 1 byte  = 43
               4 two-byte characters   =  4 × 2 bytes =  8
                                         total        = 51 bytes
              6 orthographic words

English     : 56 ASCII characters      = 56 bytes
             10 orthographic words

English at four characters per token is about 14 tokens. Azerbaijani has a byte-fallback ceiling of 51 tokens, giving a worst case of 51 / 14 = 3.6x. Compare that with the equivalent derivation for Kazakh, where the same kind of sentence produced 77 bytes and a ceiling of 5.9x. The sentences are close in meaning and structure; the byte counts differ by half again. That gap is the Cyrillic script, priced.

Both figures are derived from byte counts with the assumptions stated, not measured. The point of putting them side by side is not the absolute values — it is that the ratio between them is a property of the encoding and therefore is reliable even though neither endpoint is.

The schwa problem

ə is U+0259, LATIN SMALL LETTER SCHWA. It is the most frequent vowel in written Azerbaijani — it appears three times in the six-word sentence above — and it is not in the Latin-1 Supplement with the familiar accented vowels. It lives in the IPA Extensions block, which exists for phonetic transcription.

That matters for a reason that is easy to miss. The overwhelming majority of U+0259 in a general web crawl is not Azerbaijani prose. It is pronunciation guides, dictionary entries and linguistics papers, where the character appears inside slashes or brackets and surrounded by other IPA symbols. Whatever merges a vocabulary learned that involve U+0259 were learned from that context, and they do not correspond to Azerbaijani morpheme boundaries.

So Azerbaijani gets the worst of an in-between position. The script is cheap in bytes, which puts a low ceiling on the damage, but the single most common vowel is one the vocabulary treats as an exotic symbol, which means word-internal splits land badly and often. A word like dəyişdirilməsi is not going to be segmented at anything resembling dəyiş-dir-il-mə-si.

The dotless ı (U+0131) carries a related but distinct problem, less about cost and more about correctness: case conversion on Azerbaijani and Turkish text is locale-dependent, and a naive uppercase turns ı into I and i into I as well, collapsing two letters into one. That failure has its own page because it breaks search and deduplication rather than budgets.

What is left after the script is removed

Strip the encoding cost and Azerbaijani still does not tokenize like English, because agglutination does not care what alphabet it is written in. Azerbaijani stacks plural, possessive, case and verbal suffixes in a fixed order, and vowel harmony gives most of them two or four spellings. The result is the same combinatorial problem Kazakh has: a very large number of distinct surface forms, each individually infrequent, and a merge table that allocates by frequency.

There is one Azerbaijani-specific aggravation. Azerbaijani is written in three scripts across its speaker population — Latin in the Republic of Azerbaijan, Perso-Arabic in Iran, and Cyrillic in some older material. Whatever Azerbaijani text exists in a crawl is therefore split across three encodings that share no bytes at all. A fixed vocabulary budget that would have been thin for one script is divided three ways, and the largest speaker population writes in the script that gets the least attention in general-purpose tokenizers.

For a practical budget: on the derivation above, a 4,000-token window holding roughly 3,000 English words holds somewhere in the region of 800 to 1,500 words of Azerbaijani. That is better than Kazakh and nowhere near English, and the gap between it and Kazakh is almost entirely the alphabet.

One caution about generalising from this page to Turkish. The two languages are close, and it is tempting to reuse an Azerbaijani estimate for Turkish or the reverse. Turkish has a far larger web presence and correspondingly better merge coverage, and it does not use the schwa at all — so the two diverge on both of the variables this page identifies. Turkish has its own derivation, and borrowing either number for the other language will be wrong in the optimistic direction.

Measuring it on your own text

The claim worth testing here is the specific one: that U+0259 is where the segmentation goes wrong. This prints the tokens a word is split into, so you can look at the boundaries directly rather than at an average.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

for word in ["kitabxana", "dəyişdirilməsi", "Azərbaycan", "kitablarımızda"]:
    pieces = [enc.decode([t]) for t in enc.encode(word)]
    print(f"{word:18s} {len(pieces):2d} tokens  {pieces}")

Read the pieces, not the counts. If the splits fall inside ə or immediately either side of it while ASCII runs stay whole, the schwa argument holds for that vocabulary. If they fall at suffix boundaries, it does not, and the vocabulary has more Azerbaijani in it than this page assumes — which is a good outcome and worth knowing.