Why Ukrainian Text Costs More Tokens Than English

Ukrainian and Russian use the same Cyrillic block, the same two bytes per letter, and words of similar length. If they still tokenize differently, and they do, then the alphabet cannot be the explanation. This page derives the byte-level equivalence and then shows what is left over.

Two bytes a letter, for both languages

Cyrillic occupies U+0400 to U+04FF, entirely inside the two-byte region of UTF-8. Every letter of the Ukrainian alphabet and every letter of the Russian alphabet costs exactly two bytes. There is no three-byte penalty of the kind Bengali, Georgian or Khmer pay, and no invisible combining characters of the kind the Brahmic scripts pay for. Cyrillic is, structurally, one of the cheapest non-Latin scripts there is.

The floor is therefore about twice English per character, and that floor is identical for the two languages. Anything else is the vocabulary.

The four letters Russian does not have

Ukrainian uses four letters absent from the Russian alphabet: і (U+0456), ї (U+0457), є (U+0454) and ґ (U+0491). It also does not use ы, ъ or ё. These are all still two-byte code points, so they cost nothing extra to encode — but they matter enormously to a BPE vocabulary, because they appear in the middle of extremely common Ukrainian words and a merge learned from Russian text can never contain them.

In the sentence used below, six of the forty letters are і or ї. That is fifteen percent of the letters acting as hard barriers to any Russian-derived merge. The effect is not that those six characters are expensive; it is that they fragment the words around them. A merge covering the sequence “нженер” may exist from Russian; the Ukrainian word інженери begins with a character that stops any longer merge from starting at the word boundary, which is where BPE’s longest and most valuable merges live.

The distribution of those letters is not random either, which makes the effect worse than a fifteen percent figure implies. Ukrainian і corresponds to Russian о and е in a large class of cognates, so the words where Ukrainian and Russian look most similar are exactly the words where the differing letter sits in the middle of the stem. A vocabulary full of Russian stems is therefore maximally useless precisely where it looks most likely to help. And the letters are not the only difference: Ukrainian г is a voiced fricative written with the same code point as Russian г, while the plosive is the separate ґ (U+0491), one of the least frequent letters in the Unicode Cyrillic code chart and near-certainly absent from any merge longer than itself.

Deriving the multiplier, and the gap

The sentence is “Ukrainian engineers opened new possibilities”, as Українські інженери відкрили нові можливості. and Украинские инженеры открыли новые возможности.

Ukrainian
  code points ............ 45  (40 letters + 4 spaces + 1 full stop)
  UTF-8 bytes ............ 85   (40 x 2, plus 5 ASCII)

Russian
  code points ............ 46  (41 letters + 4 spaces + 1 full stop)
  UTF-8 bytes ............ 87   (41 x 2, plus 5 ASCII)

  Difference: 2 bytes, or 2.3%. At the encoding level these
  two sentences cost the same.

English "Ukrainian engineers opened new possibilities."
  characters ............. 45
  tokens (assumption: ~4 chars/token for English) ..... ~11

Ceiling (1 token per UTF-8 byte)
  Ukrainian  85 / 11 = 7.7x English
  Russian    87 / 11 = 7.9x English

Band    (assume Russian resolves at 3.5 bytes/token because the
         vocabulary has whole-word Russian merges, and Ukrainian
         at 2.5 because it does not)
  Russian    25 tokens -> 2.3x     Ukrainian  34 tokens -> 3.1x

The two bytes-per-token assumptions in that last block are the entire claim of this page, and they are assumptions, not measurements. The point is structural: since the byte counts are within 2.3% of each other, any observed difference larger than that in token count is caused by the vocabulary and by nothing else. Measure it:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

uk = "Українські інженери відкрили нові можливості."
ru = "Украинские инженеры открыли новые возможности."

for lang, s in (("uk", uk), ("ru", ru)):
    ids = enc.encode(s)
    bpt = len(s.encode("utf-8")) / len(ids)
    print(lang, len(s.encode("utf-8")), "bytes", len(ids), "tokens",
          round(bpt, 2), "bytes/token")

# Where does Ukrainian fragment? Print the pieces.
print([enc.decode_single_token_bytes(i).decode("utf-8", "replace")
       for i in enc.encode(uk)])

That last line is worth running. The fragments show you the shape of the problem directly: Russian tends to come back as recognisable words and stems, Ukrainian as two- and three-letter pieces with the distinctively Ukrainian letters sitting alone.

Merges are allocated by frequency, not by alphabet

A BPE vocabulary is built by repeatedly merging the most frequent adjacent pair in a training corpus, as described in Sennrich, Haddow and Birch’s 2015 paper introducing subword units. Nothing in that procedure knows what a language is. It knows what byte sequences are frequent. Russian is one of the larger non-English languages on the crawled web; Ukrainian is a fraction of its size in most crawls. A vocabulary trained on such a corpus allocates its Cyrillic budget accordingly, and the result is that the alphabet is shared while the merges are not.

Bulgarian, Serbian and Macedonian sit in the same position with respect to Russian, each with its own orthographic quirks that break Russian merges — see Bulgarian for the South Slavic version of the argument.

One consequence of this is easy to miss and matters for planning. A tokenizer vocabulary is frozen at the moment it is built, and it is built once per model generation rather than continuously. Growth in the amount of Ukrainian text on the web has no effect at all on a model whose vocabulary was fixed before that text existed — it only shows up when a vendor ships a new encoding, and vendors ship new encodings rarely. So the gap described on this page is not a moving target that quietly closes. It closes in steps, at announcements, and the correct response is to re-measure when a provider changes tokenizer rather than to assume gradual improvement.

What this changes in practice

• Do not size a Ukrainian budget from a Russian benchmark. The two are close enough that people substitute one for the other, and the substitution is wrong in the direction that causes silent truncation rather than wasted headroom.
• Watch for Russian bleeding into Ukrainian output. A model whose Cyrillic competence is mostly Russian will drift into Russian word forms under pressure, particularly at low temperature on a Ukrainian prompt with an English system message. Set the output language explicitly and check it, rather than assuming the script implies the language.
• Proper names are the worst case. Ukrainian place and personal names are exactly where the і/ї letters cluster and exactly where a Russian-derived merge is most likely to produce the Russian spelling. This is a correctness problem as much as a cost one; see transliterating Ukrainian names.
• Apostrophes matter. Ukrainian uses an apostrophe as a letter-level separator. Whether it arrives as U+2019, U+02BC or ASCII ' changes the byte sequence and therefore the merges. Normalise it before indexing or you will fragment the same word two different ways.