Why Mongolian Text Costs More Tokens Than English

Mongolian written in Cyrillic looks, byte for byte, like a Slavic language. It has the same two-byte code points, the same alphabet shape, the same absence of combining marks. It tokenizes nothing like Russian, and the gap between those two facts is the most direct demonstration available that byte-pair merges are learned from words and not from alphabets.

Cyrillic, plus two letters Russian does not have

Mongolian has been written in Cyrillic in Mongolia since the 1940s. The alphabet is the Russian one plus two letters: ө (U+04E9) and ү (U+04AF), which write the front rounded vowels that Mongolian has and Russian does not. Both live in the Cyrillic block, so both cost two UTF-8 bytes like every other Cyrillic letter. There is no encoding penalty relative to Russian whatsoever.

But those two letters are extremely frequent in Mongolian, because vowel harmony means that a word containing a front rounded vowel tends to contain several. In the sentence used below, five of thirty-nine letters are ө or ү. Each occurrence is a code point that is rare in the corpus overall, and therefore a point at which no Russian-derived merge can extend. This is the same barrier effect that Ukrainian’s і and ї produce, described under Ukrainian — but Ukrainian at least shares a large Slavic word stock with Russian and Mongolian shares none at all.

Why a shared alphabet transfers nothing

A BPE vocabulary contains byte sequences, ranked by how often they occurred in training. The valuable entries are long: whole words, whole stems, common suffixes, frequent word-initial sequences preceded by a space. Those entries are specific to a language’s lexicon.

Russian and Mongolian are not related. Russian is Indo-European; Mongolian is Mongolic. They share no inherited vocabulary, no inflectional endings, no derivational morphology. What Mongolian inherits from Russian in the vocabulary is limited to two things: the 33 single-letter entries for the Cyrillic block, and whatever short two- or three-letter sequences happen to coincide by chance. There are also a few hundred Russian loanwords in modern Mongolian, mostly technical, which do transfer — and which are a rounding error in running text.

There is a second-order problem that shows up in user-generated Mongolian and costs more than it should. Because ө and ү are awkward to type on keyboards configured for Russian, they are routinely substituted with visually similar characters — Latin o and y, Cyrillic о and у, or the Cyrillic ѳ. Each substitution produces a different byte sequence for the same intended word, so one word indexes several ways, none of them matching the others, and a mixed-script form fragments harder than either script alone would. The relevant Unicode block assignments are set out in the Unicode Cyrillic code chart; a confusable-folding pass on ingest is worth more here than any tokenizer choice.

The general principle is worth stating because it applies far beyond Mongolian: script coverage and language coverage are different things. A tokenizer that handles Cyrillic perfectly can still be near-worthless for a Cyrillic-written language it never saw. Kazakh sits in the same position, with its own extra Cyrillic letters and its own unrelated Turkic lexicon; see Kazakh.

Vowel harmony and long suffix chains

Mongolian is agglutinative. Case, number, possession and a rich set of verbal categories attach as ordered suffixes, and vowel harmony means each suffix has front and back variants selected by the vowels of the stem. So a single stem generates many surface forms, and each form exists in at least two vowel-harmonic spellings.

For a merge table this is the worst combination available: a large space of surface forms, each individually infrequent, each written with characters that are themselves comparatively rare. It is the same structural problem Turkish has, without Turkish’s substantially larger web corpus and without Turkish’s mostly-ASCII Latin script to soften it.

Deriving the multiplier

The sentence is Монгол хэл нь өөрийн гэсэн үг бүтээх дүрэмтэй. — “Mongolian has its own word-formation rules.”

Mongolian (Cyrillic)
  code points ............ 46   (38 Cyrillic letters, 7 spaces, 1 '.')
  UTF-8 bytes ............ 84   (38 x 2, plus 8 ASCII)
  letters outside the Russian alphabet: 5 of 38  (ө x2, ү x3)

Compare Russian, from the Ukrainian page in this cluster:
  a 41-letter Russian sentence is 87 bytes. Mongolian's byte cost
  per letter is identical. Any token difference is vocabulary.

English "Mongolian has its own word-formation rules."
  characters ............. 43
  tokens (assumption: ~4 chars/token for English) ..... ~11

Ceiling (1 token per UTF-8 byte)
  84 / 11 = 7.6x English

Band    (assume 2 bytes/token, one token per Cyrillic letter --
         the realistic case when only single-letter entries transfer)
  84 / 2 = 42 tokens  ->  42 / 11 = 3.8x English

Band    (assume 3.5 bytes/token, i.e. real Mongolian merges exist)
  84 / 3.5 = 24 tokens  ->  24 / 11 = 2.2x English

Derived, not measured. The test that proves the no-transfer claim is to run Mongolian and Russian through the same encoding and compare bytes per token, since bytes per letter are identical by construction:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

mn = "Монгол хэл нь өөрийн гэсэн үг бүтээх дүрэмтэй."
ru = "Монгольский язык имеет собственные правила словообразования."

for lang, s in (("mn", mn), ("ru", ru)):
    ids = enc.encode(s)
    b = len(s.encode("utf-8"))
    print(lang, b, "bytes", len(ids), "tokens",
          round(b / len(ids), 2), "bytes/token")

# Same alphabet, same bytes per letter. If bytes/token differs
# substantially, the difference is merges and nothing else.
print([enc.decode_single_token_bytes(i).decode("utf-8", "replace")
       for i in enc.encode(mn)])

The printed fragments are the evidence. Russian typically comes back as words and stems; Mongolian typically comes back as two- and three-letter pieces, with ө and ү standing alone.

Two scripts, one language

Mongolian is also written in the traditional Mongolian script, the vertical script used in Inner Mongolia and increasingly taught in Mongolia itself. That script occupies U+1800 to U+18AF, which is above the U+0800 boundary and therefore three bytes per code point rather than two, and it uses the free variation selectors U+180B to U+180D to control contextual letterforms — more invisible code points, each costing three bytes.

• The two scripts are not interchangeable for a model. They are different Unicode blocks with different byte sequences and almost certainly very different vocabulary coverage. Whatever you measure for Cyrillic Mongolian tells you nothing about traditional Mongolian.
• Traditional Mongolian is worse on every axis. Three bytes per code point, invisible variation selectors, a smaller digital corpus, and vertical layout that most rendering stacks handle badly.
• Detect the script before you route. A language detector that returns “Mongolian” without a script tag is not enough information to pick a model or size a budget. Check the Unicode block of the text itself.
• Scanned material is mostly vertical. If your Mongolian corpus is historical, expect the traditional script and expect the vertical reading order to break OCR; see OCR of vertical Mongolian script.