Why Russian Text Costs More Tokens Than English

Russian is the best-served non-Latin language in most tokenizers, and it still costs roughly twice English. Half of that is Unicode arithmetic and half is grammar, and only the grammar half varies with what you are writing.

Two bytes a letter, and what that buys

The Cyrillic block runs from U+0400 to U+04FF, inside the range UTF-8 encodes in two bytes. There are no combining marks in ordinary Russian orthography, no joiners, and no decomposition question: й and ё are precomposed single code points in normal use. Compared with the rest of this cluster, Russian is structurally simple, which is why the interesting variable is elsewhere.

Эта функция использует много памяти компьютера.
"This function uses a lot of computer memory."

Forty-one Cyrillic letters at two bytes each, five spaces and a full stop at one byte each: exactly 88 bytes. The English is 44 bytes, near 11 tokens. So Russian costs exactly twice the bytes of its English translation here, and since byte-level BPE cannot exceed one token per byte, 88 is the ceiling.

Derive the middle. Assume that high-frequency Russian words and word-initial fragments — это, что, не, and the common prefixes — have earned merges, while longer content words break into two or three pieces. That places this sentence around 14 to 22 tokens against the English 11, so a derived multiplier of roughly 1.3× to 2×. That is the good case, and Russian generally sits at the friendly end of the non-Latin range for exactly one reason: there is a great deal of Russian on the public web, so the corpus that built the merge table contained a lot of it.

Six cases against the merge table

Russian is fusional. A noun inflects for six cases in two numbers, so a single lemma appears in up to twelve written forms, and adjectives agree with it in case, number and gender, multiplying again. компьютер, компьютера, компьютеру, компьютером, компьютере, компьютеры, компьютеров are all the same word.

For a merge table, this is a budget problem. English computer has essentially two surface forms and can be earned as one token with a leading space. Russian has to spend a slot on each inflected form it wants to represent whole, competing against every other word in every other language. It cannot afford all of them, so it typically keeps the stem and lets the endings be separate tokens. The result is a systematic extra token or two per content word that has nothing to do with byte width and everything to do with morphology.

The comparison that makes this legible is with an isolating language. English marks the same relations with separate words — prepositions and word order — and those separate words are themselves frequent enough to be single cheap tokens. Russian folds the relation into the word ending, which produces a longer, rarer string. Both languages pay for grammar; English pays in short common tokens and Russian pays in fragmenting long ones.

Russian prefixes push in the other direction and are worth knowing about because they are the one place the merge table gets ahead. Russian verbs take a small, closed set of highly productive prefixes — по-, при-, пере-, вы-, за-, раз- — and because that set is small and enormously frequent, those prefixes are exactly the kind of sequence BPE learns first. A prefixed verb therefore often costs a prefix token plus a stem token rather than falling apart into letter pairs. The endings are the expensive part; the beginnings are not.

There is a generation-side consequence that catches people out. Output tokens are priced several times higher than input tokens by most providers, so a multiplier of two on Russian output costs more than the same multiplier on a Russian prompt. And a max_tokens value chosen by looking at English answers will cut Russian answers off part way through, typically mid-word, because the answer takes roughly twice the tokens to say the same thing. Set output limits from a Russian sample, not an English one.

The practical form of the inflection penalty appears in retrieval, where a query in the nominative does not string-match a document in the genitive, and in generation, where agreement errors are the characteristic failure — see case agreement errors in generated Russian.

Being Cyrillic is not the same as being Russian

This is the part that generalises badly and is worth stating plainly. The favourable multiplier derived above belongs to Russian, not to Cyrillic. Ukrainian adds і, ї, є and ґ; Serbian adds ђ, ј, љ, њ, ћ and џ; Macedonian adds ѓ and ќ; Bulgarian shares the Russian inventory but has completely different high-frequency word forms. Every one of those is two bytes and every one of them is far rarer in a training corpus than the Russian letters around it.

The consequence is that a language sharing Russian’s script can have a materially worse multiplier, because the merges available are Russian merges that mostly do not fire on its words. Do not reuse a Russian figure for another Cyrillic language; derive it separately, as the Ukrainian page does.

Measuring the inflection penalty

The useful measurement is not the whole-document ratio but the per-inflected-form one, because that is the part that varies by genre. Legal and bureaucratic Russian is dense in oblique cases; dialogue is not.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

forms = ["компьютер", "компьютера", "компьютеру",
         "компьютером", "компьютере", "компьютеров"]

for w in forms:
    s = " " + w
    print(w, "bytes", len(s.encode("utf-8")), "tokens", len(enc.encode(s)))

print("english", len(enc.encode(" computer")), "token(s)")

What you are looking for is the spread. If the nominative costs two tokens and the instrumental costs four, the merge table has the stem and not the endings, and your cost per document scales with how oblique your text is. If all six cost the same, the tokenizer is splitting on the stem boundary cleanly and the penalty is a flat one token per word rather than a variable one.

What it changes in practice

• Budget chunks at roughly half the English character count. A 4,000-token chunk holding about 16,000 English characters holds something closer to 6,000–9,000 Cyrillic characters on the derivation above.
• Lemmatise before keyword matching, not before embedding. Case endings defeat exact match, but they carry real information that an embedding model uses; stripping them helps BM25 and hurts semantic search.
• Watch for Latin lookalikes. Cyrillic а, е, о, р, с, у and х are visually identical to Latin letters and are different code points. Mixed-script words tokenise badly and match nothing, and they arrive from OCR and from copy-paste far more often than anyone expects.
• ё is frequently written as е. The two are distinct code points and the substitution is conventional in Russian typing, so the same word can arrive in two spellings with two different token counts.