Why Serbian Text Costs More Tokens When Written in Cyrillic

Serbian is written in Cyrillic and in Latin, both officially, with a strict letter-for-letter correspondence between them. Nothing about the language changes when you switch. That makes it the only case in this cluster where the token cost of a script can be derived without comparing two different languages and hoping they are comparable.

One language, two orthographies

Serbian orthography is phonemic in both alphabets and the mapping is bidirectional: љ is lj, њ is nj, џ is dž, ч is č, and so on for the whole alphabet. A converter is a lookup table, not a transliteration system with judgement calls in it — unlike the various Cyrillic romanisation standards, which disagree with each other precisely because the languages they serve do not have this property.

So a Serbian document has an exact twin. Same words, same word order, same meaning, different bytes. Everything below follows from writing out both.

One practical wrinkle before the arithmetic: real Serbian documents are frequently mixed. A page can carry Cyrillic body text with Latin headings, Latin brand names inside Cyrillic sentences, or a form whose labels and whose user-entered values are in different alphabets. That means a per-document average of anything — bytes per character, tokens per word — is a blend of two populations rather than a measurement of either, and it is the reason the script below reports the two spellings separately instead of summarising the file.

The same sentence, derived twice

Cyrillic : Он живи у Београду и говори српски.
Latin    : On živi u Beogradu i govori srpski.

Cyrillic : 28 Cyrillic letters      = 28 × 2 bytes = 56
           6 spaces + 1 full stop   =  7 × 1 byte  =  7
                                      total        = 63 bytes

Latin    : 27 ASCII letters         = 27 × 1 byte  = 27
           1 two-byte letter (ž)    =  1 × 2 bytes =  2
           6 spaces + 1 full stop   =  7 × 1 byte  =  7
                                      total        = 36 bytes

English  : He lives in Belgrade and speaks Serbian.
           39 ASCII characters      = 39 bytes

The English gloss at roughly four characters per token is about ten tokens. The Cyrillic spelling has a byte-fallback ceiling of 63 tokens, the Latin spelling of 36. Derived worst cases: 63 / 10 = 6.3x for Cyrillic and 36 / 10 = 3.6x for Latin. These are ceilings computed from byte counts, not measurements, and no tokenizer was run to produce them.

The ratio between the two is the durable part. Cyrillic Serbian is about 1.75 times the bytes of Latin Serbian for identical content, and that figure comes from the encoding rather than from any vocabulary. In practice the gap in real token counts is usually wider than the byte ratio, not narrower, because the Latin spelling also benefits from merges learned on Croatian, Bosnian, Slovenian and every other Latin-script language, while the Cyrillic spelling competes for attention with Russian.

The digraph trade

The letter-for-letter mapping is not character-for-character. Three Serbian Cyrillic letters correspond to two-letter Latin sequences, and they run in opposite directions on the two things you might count:

љ  →  lj      1 char, 2 bytes   →   2 chars, 2 bytes
њ  →  nj      1 char, 2 bytes   →   2 chars, 2 bytes
џ  →  dž      1 char, 2 bytes   →   2 chars, 3 bytes  (ž is two bytes)

људи  →  ljudi     4 chars,  8 bytes   →   5 chars, 5 bytes
џак   →  džak      3 chars,  6 bytes   →   4 chars, 5 bytes

Cyrillic Serbian is consistently shorter in characters and consistently longer in bytes. Anyone who sizes a field, a prompt or a retrieval chunk by character count and then reasons about tokens from that character count will get Serbian wrong in the direction that overflows — the Cyrillic text looks shorter and costs more.

There is a second-order effect on the Latin side worth knowing about. Because lj, nj and dž are ordinary two-letter sequences in Latin, a tokenizer will happily merge across the boundary between a digraph and the letter next to it, producing splits that cut a single phoneme in half. That does not cost extra tokens; it costs meaningful boundaries, which is a retrieval and embedding problem rather than a billing one.

What follows from the choice

If you control the orthography of the text you send — system prompts, few-shot examples, a knowledge base you author — writing it in Latin is a real and substantial cost reduction with no change in meaning. On the derivation above the input side of a Serbian prompt costs roughly 40 percent less.

Work that through on something concrete. A 2,000-word Serbian system prompt is around 13,000 characters. In Latin that is close to 13,500 bytes; in Cyrillic it is around 26,000. If the prompt is sent on every request and the merge coverage is similar in both directions, the Cyrillic version carries something in the region of twice the input tokens for identical instructions — on a prompt that is resent with every call, forever, and that no user ever sees. That is the single cheapest change available to a Serbian-language product, and it requires no engineering at all beyond running the text through a lookup table once.

If you do not control it, do not convert user text silently. Three things break:

• Proper nouns and foreign words. Serbian Latin keeps some foreign names in their original spelling in contexts where Cyrillic transcribes them phonetically. A round trip does not always return the original.
• The output language. A model given a Latin prompt usually answers in Latin. If your users read Cyrillic, converting the input changes what they receive, and converting the output back is another lossy pass.
• Stored text and search. If you index the converted form and store the original, a query in the other alphabet matches nothing. Pick one normalisation for the index and apply it to queries too.

The related case is Macedonian, which shares much of the Cyrillic inventory but has no Latin standard to escape to, and Russian, where the same two-byte arithmetic applies against a very much larger training share.

Measuring both, and converting safely

Since the two spellings are a lookup table apart, you can measure the real gap on your own corpus rather than trusting the derivation.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

CYR2LAT = {
    "љ": "lj", "њ": "nj", "џ": "dž", "ђ": "đ", "ћ": "ć", "ч": "č",
    "ж": "ž", "ш": "š", "а": "a", "б": "b", "в": "v", "г": "g",
    "д": "d", "е": "e", "з": "z", "и": "i", "ј": "j", "к": "k",
    "л": "l", "м": "m", "н": "n", "о": "o", "п": "p", "р": "r",
    "с": "s", "т": "t", "у": "u", "ф": "f", "х": "h", "ц": "c",
}

def to_latin(s):
    out = []
    for ch in s:
        lower = ch.lower()
        rep = CYR2LAT.get(lower, ch)
        out.append(rep.capitalize() if ch.isupper() else rep)
    return "".join(out)

cyr = open("serbian_cyrillic.txt", encoding="utf-8").read()
lat = to_latin(cyr)

c, l = len(enc.encode(cyr)), len(enc.encode(lat))
print(f"cyrillic {c} tokens, latin {l} tokens, ratio {c / l:.2f}")

The table above is deliberately incomplete — it covers the lowercase letters and leaves out the ones your text may not contain — because a conversion you paste in without reading is exactly how the proper-noun problem above reaches production. Use it to measure, and use a maintained library if you intend to convert anything you keep.