Why Bulgarian Text Costs More Tokens Than English

Bulgarian and Russian are written in the same alphabet, and a Bulgarian sentence and its Russian translation take almost exactly the same number of bytes. They do not take the same number of tokens. That gap is the clearest available demonstration that a script penalty and a language penalty are two different things.

The script is not the variable

Every Cyrillic letter sits below U+0800 and therefore takes two bytes in UTF-8. That is true of Russian, Bulgarian, Macedonian, Serbian, Ukrainian and Kazakh alike. If the script were the whole story, all six would carry the same multiplier against English, and they do not.

What differs is how much of each language a vocabulary saw during training. Russian is among the largest non-English languages on the public web by a wide margin; Bulgarian has roughly eight million speakers and a correspondingly small footprint. A byte-pair vocabulary allocates merges by frequency across the whole training mix, so the Cyrillic merges it holds are overwhelmingly Russian-shaped: Russian stems, Russian inflectional endings, Russian orthographic conventions.

Bulgarian shares a great deal of that. The two languages have common Slavic roots and a large stock of cognates, so a Russian-derived merge often fires usefully on Bulgarian. But it fires on Russian boundaries, and Bulgarian word endings are not Russian word endings, so the tail of every Bulgarian word tends to fragment where the head did not.

There is a second, less obvious source of divergence. The two languages do not spell their shared inheritance identically. Bulgarian uses ъ as a full vowel where Russian uses the same letter as a silent hard sign, and Bulgarian has no ы and no э at all. So a word the two languages genuinely share often arrives as a different byte sequence, and the Russian merge for it does not fire even though a reader would call them the same word. Recognisable to a person and invisible to a merge table is the normal situation for closely related languages, which is why relatedness buys far less tokenizer efficiency than it intuitively should.

Deriving both multipliers

Bulgarian : В библиотеката има много книги на български език.
Russian   : В библиотеке много книг на болгарском языке.
English   : There are many Bulgarian-language books in the library.

Bulgarian : 41 Cyrillic letters   = 41 × 2 bytes = 82
            7 spaces + full stop  =  8 × 1 byte  =  8
                                    total        = 90 bytes

Russian   : 37 Cyrillic letters   = 37 × 2 bytes = 74
            6 spaces + full stop  =  7 × 1 byte  =  7
                                    total        = 81 bytes

English   : 55 ASCII characters   = 55 bytes  ≈ 14 tokens

Byte-fallback ceilings: 90 tokens for Bulgarian against 81 for Russian, or 6.4x and 5.8x against the English baseline of about fourteen tokens. Those ceilings are within ten percent of each other, and they are derived arithmetic rather than measured counts.

The ceilings are what the script alone can explain. Any larger gap between the two languages in a real token count — and there generally is one — cannot come from the encoding, because the encoding says they are nearly the same. It comes from the merge table, and the merge table is a record of how much Russian and how little Bulgarian went into it.

The suffix Russian does not have

Bulgarian is the outlier of the Slavic family in a way that works against it here. It lost the noun case system almost entirely, which makes its grammar simpler than Russian’s, and it gained a postposed definite article, which makes its word forms longer than Russian’s. библио тека is a library; библиоте ката is the library. The definiteness English carries in a separate three-letter word is glued to the end of the Bulgarian noun.

This is exactly the shape a Russian-trained merge table handles worst. The stem is recognisable and probably merges well, because Russian has the same root. The article suffix — one of a small set including -та, -то, -те, -ът, -ят — does not exist in Russian at all. Every definite noun in a Bulgarian document therefore ends in a fragment the vocabulary has no good merge for, and definite nouns are not rare.

The same logic explains why Bulgarian’s plural and agreement behaviour is a separate practical problem from its cost; the counting forms and the numeric agreement rules are covered in the Slavic plural categories page, and they are the thing that breaks generated Bulgarian text rather than the thing that makes it expensive.

What this costs a Bulgarian product

Work the derivation forward. If Bulgarian text lands in the region of three to five times English tokens for the same content — which is what the bounded range and the merge-table argument together suggest — then three things change for a product serving Bulgarian users:

• The context window shrinks in real terms. A window that holds thirty pages of English documentation holds six to ten of the Bulgarian translation. A retrieval system tuned on English chunk sizes will truncate.
• Output limits bite earlier. A max_tokens of 500 that comfortably produces an English paragraph produces a fraction of one in Bulgarian, and the truncation arrives as a finish_reason of length rather than as an error you would notice in testing.
• Per-user cost diverges by locale. The same feature, the same prompt template and the same model cost several times more per Bulgarian user than per English one. If your unit economics are computed on an English average, they are wrong for every non-English market and wrong by a different amount in each.

The Macedonian case is worse for a mechanically different reason, and the two pages are deliberately separate: Macedonian’s problem starts before training, in how corpora are filtered.

Measuring it on your own text

The claim to test is the specific one about the article suffix. Take nouns in both forms and look at where the splits land.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

pairs = [
    ("библиотека", "библиотеката"),   # library / the library
    ("книга",      "книгата"),        # book    / the book
    ("град",       "градът"),         # city    / the city
]

for bare, definite in pairs:
    b = [enc.decode([t]) for t in enc.encode(bare)]
    d = [enc.decode([t]) for t in enc.encode(definite)]
    print(f"{bare:14s} {len(b)}  {b}")
    print(f"{definite:14s} {len(d)}  {d}")
    print(f"  cost of definiteness: +{len(d) - len(b)} tokens\n")

If adding the article costs one token, the vocabulary has learned the suffix and Bulgarian is better represented than this page assumes. If it costs two or three, the suffix is being spelled out in fragments, which is the predicted behaviour and the reason a Bulgarian document costs more than a Russian one of the same byte length.