Why Macedonian Text Costs More Tokens Than English

Macedonian carries the same two-byte Cyrillic arithmetic as every other Cyrillic language, and a worse outcome than most of them. The reason is not the script and not the grammar. It is that a large share of Macedonian text on the web is labelled as something else before anybody trains anything on it.

Deriving the multiplier

Macedonian : Во библиотеката има многу книги на македонски јазик.
English    : There are many Macedonian-language books in the library.

Macedonian : 44 Cyrillic letters   = 44 × 2 bytes = 88
             7 spaces + full stop  =  8 × 1 byte  =  8
                                     total        = 96 bytes

English    : 56 ASCII characters   = 56 bytes  ≈ 14 tokens

The byte-fallback ceiling is 96 tokens against roughly fourteen for English, so the derived worst case is 96 / 14 = 6.9x. The floor, if every one of the eight words were a single learned token, is 8 / 14, which is below one. Neither endpoint is realistic on its own; both are arithmetic from the byte count, with the four characters per token English baseline that OpenAI publishes on its tokenizer page.

Compare the same derivation for Bulgarian: 90 bytes for a sentence that means the same thing, a ceiling of 6.4x. The two languages are within a few percent of each other on bytes, as you would expect from two closely related South Slavic languages in the same alphabet. Whatever separates them in practice is not encoded in the sentence.

The shortfall starts before training

Web-scale corpora are built by crawling, then filtering, then deduplicating, and the filtering stage assigns a language label to every document. That label decides whether the document is kept, which language bucket it lands in, and what proportion of the final mix each language gets. Language identification is therefore not a post-processing detail; it is the gate.

Macedonian is one of the hardest cases for that gate. It is mutually intelligible in large part with Bulgarian, shares most of its orthography with both Bulgarian and Serbian Cyrillic, and has around two million speakers producing a correspondingly small volume of text. A classifier trained with Bulgarian and Serbian as much larger classes will assign a short Macedonian document to one of them a substantial fraction of the time, and a short document is the common case on the web.

Three failure modes follow, and they compound:

• Mislabelled. The document survives but counts toward Bulgarian or Serbian, so Macedonian’s measured share of the corpus is understated and any sampling that upweights small languages does not upweight this one.
• Dropped for low confidence. Pipelines commonly discard documents where the classifier is unsure. Macedonian is systematically the case the classifier is unsure about, so the filter removes it preferentially rather than at random.
• Deduplicated against its neighbours. Near-duplicate detection across a Cyrillic pool can treat a Macedonian text and a close Bulgarian one as the same document and keep the more common variant.

None of this is about the tokenizer. It happens upstream, and the tokenizer inherits the result: a merge table with essentially no Macedonian-specific evidence in it. The general problem of distinguishing closely related languages has its own page, and Macedonian is its most consequential instance.

The letters that carry the identity

Macedonian’s alphabet differs from Bulgarian’s in a small number of letters, and those letters are the ones a classifier could use if the document is long enough to contain them:

• ј (U+0458), љ (U+0459), њ (U+045A), џ (U+045F) — shared with Serbian, absent from Bulgarian.
• ѓ (U+0453) and ќ (U+045C) — Macedonian’s own, where Serbian writes ђ and ћ and Bulgarian writes neither.
• ѕ (U+0455) — the dz letter, rare in running text and rarer still in any other Cyrillic orthography.

These are charted by the Unicode Consortium in the Cyrillic block, in the U+0450–U+045F range that holds the non-Russian Cyrillic letters. Their rarity is the trap: a sentence can be unambiguously Macedonian to a reader and contain none of them, which is precisely when a classifier guesses Bulgarian.

For the tokenizer the same rarity has a direct cost. ѓ and ќ appear in essentially no other language’s text, so no merge involving them was ever learned from anywhere else. Words containing them fall to byte fallback more completely than the rest of a Macedonian document does, and those words are not obscure — кќе and веѓе are everyday vocabulary.

What thin representation costs beyond tokens

Token cost is the measurable part of a wider problem, and it is worth being explicit that they travel together rather than treating the bill as the whole issue. A language whose text was filtered out of the corpus is a language the model has less of everywhere: fewer merges in the vocabulary, less signal in the embedding space, and a higher rate of the failure where a model answers a Macedonian prompt in Bulgarian because that is the nearest thing it knows well.

The practical consequence for a budget: on the derivation above a 4,000-token context holds roughly 700 to 1,300 words of Macedonian against about 3,000 of English. If you are building retrieval over Macedonian documents, size the chunks in tokens, and expect the chunk-to-content ratio to be closer to Serbian Cyrillic’s than to anything Latin-script.

Measuring it on your own text

Two things are worth measuring, and the second is the one people skip. The first is the token multiplier. The second is whether your own pipeline is committing the upstream error this page describes — if you run language detection anywhere before indexing, check what it says about your Macedonian documents.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

# 1. the multiplier, per document
for line in open("macedonian.txt", encoding="utf-8"):
    line = line.strip()
    if not line:
        continue
    b, t = len(line.encode("utf-8")), len(enc.encode(line))
    print(f"{b/t:4.2f} bytes/token   {line[:50]}")

# 2. the words that fall hardest — those containing ѓ, ќ, ѕ
for word in ["куќа", "вреќа", "ѓавол", "ѕвезда", "книга"]:
    print(word, len(enc.encode(word)), "tokens")

A bytes-per-token close to 1.0 means byte fallback. If the second list shows the words containing ќ, ѓ and ѕ costing noticeably more per character than книга, which is spelled the same way in Bulgarian and Russian, the argument on this page is visible in your own data.