Why Polish Text Costs More Tokens Than English

Polish never hits byte fallback. Every character it uses is either ASCII or a well-known Latin Extended code point, and Polish is one of the better-represented European languages on the crawled web. It still costs roughly two to three times English, and the reason is visible in one sentence if you count the right thing.

Latin script, and still not cheap

It is worth saying at the start what Polish is not paying. It is not paying a three-byte encoding, like Bengali. It is not falling back to raw bytes, which is what happens to a script with no merges at all. Twenty-six of its thirty-two letters are plain ASCII, one byte each, sitting in the same part of the vocabulary as English.

What Polish pays instead is a subtler tax, and it is more instructive than the dramatic cases because it shows what BPE actually optimises. A merge table does not reward a language for using a familiar alphabet. It rewards a language for producing byte sequences that recur.

Eight two-byte characters in one sentence

Polish has nine letters with diacritics: ą, ć, ę, ł, ń, ó, ś, ź and ż. The ogonek letters ą and ę sit in Latin Extended-A, as do ć, ł, ń, ś, ź and ż per the Unicode Latin Extended-A code chart; ó is in Latin-1 Supplement. Every one of them is two UTF-8 bytes rather than one. That is a fifty percent surcharge on those characters and nothing on the rest, which sounds trivial.

The cost is not the extra byte. The cost is that the extra byte falls in the middle of words. Consider książek, the genitive plural of “book”. To a byte-level tokenizer that is k, s, i, two bytes for ą, two bytes for ż, then e, k — nine bytes, with two two-byte islands that no English-derived merge covers, sitting in positions three and five of a seven-character word. There is no long merge that spans them unless the vocabulary learned this specific Polish word. The ASCII neighbours are individually cheap and collectively useless, because BPE’s value comes from long merges and every diacritic is a place a long merge has to stop.

The sentence used below contains exactly eight such characters across fifty-two characters of text, which is how its byte count comes to sixty rather than fifty-two. Three of them are in a single word.

Seven cases and what they do to a merge table

Polish is fusional, with seven cases, three genders in the singular and a masculine-personal versus non-masculine-personal split in the plural, and adjectives that agree with all of it. A single noun has on the order of a dozen distinct surface forms and a single adjective more. That multiplies the number of byte sequences a vocabulary would need to cover the language well, and divides the frequency of each one.

Unlike Tamil or Turkish, Polish does not build long agglutinated chains — a fusional language packs several grammatical categories into one short ending rather than stacking one suffix per category. So Polish word forms stay short. What they do instead is proliferate: książka, książki, książce, książkę, książką, książek, książkom, książkami, książkach. Nine forms of one noun, each carrying the same two diacritic islands, each individually less frequent than the English “book” or “books”. The same mechanism drives Hungarian and Finnish, in their case with agglutination rather than fusion.

Deriving the multiplier

The sentence is Wczoraj wszyscy zdążyliśmy przeczytać sześć książek. — “Yesterday we all managed to read six books.”

Polish
  characters ............. 52
  UTF-8 bytes ............ 60   (44 x 1 byte, plus 8 diacritics x 2 bytes)
  diacritics ............. 8    (ą,ż,ś in zdążyliśmy; ć in przeczytać;
                                 ś,ć in sześć; ą,ż in książek)

English "Yesterday we all managed to read six books."
  characters ............. 43
  tokens (assumption: ~4 chars/token for English) ..... ~11

Ceiling (1 token per UTF-8 byte -- unreachable for Latin script,
         included only to bound the answer)
  60 / 11 = 5.5x English

Band    (assume Polish resolves at 2.5 bytes/token: shorter merges
         than English's ~4, because diacritics break them)
  60 / 2.5 = 24 tokens  ->  24 / 11 = 2.2x English

Band    (assume 3 bytes/token on a vocabulary with good Polish coverage)
  60 / 3 = 20 tokens  ->  20 / 11 = 1.8x English

Derived, not measured. Polish is one of the languages where the band is narrow and the answer is genuinely modest — roughly two times, not the six or ten times a three-byte script pays. The test that isolates the diacritic effect is to strip them and re-measure:

import tiktoken, unicodedata
enc = tiktoken.get_encoding("o200k_base")

pl = "Wczoraj wszyscy zdążyliśmy przeczytać sześć książek."

def fold(s):
    # NFD splits a letter from its combining mark; drop the marks.
    # Note this does NOT handle ł, which has no decomposition.
    d = unicodedata.normalize("NFD", s)
    return "".join(c for c in d if not unicodedata.combining(c))

for label, s in (("polish", pl), ("folded", fold(pl))):
    ids = enc.encode(s)
    print(label, len(s.encode("utf-8")), "bytes", len(ids), "tokens")

The comment in that script is not a footnote. The stroked l has no canonical decomposition in Unicode, so NFD leaves it alone and any accent-stripping routine built on NFD silently misses it. That single fact causes a large share of the Polish text-processing bugs that look like tokenizer problems and are not.

The stripping temptation, and why it fails

The folded string in that script costs fewer tokens. It is also not Polish. Stripping diacritics changes meaning, sometimes catastrophically: łaska is grace and laska is a walking stick; sad is an orchard and sąd is a court. A model asked to reason over stripped Polish is reading a systematically ambiguous text, and an embedding built from stripped Polish will not match a correctly-written query.

• Normalise, do not fold. NFC on input so that a precomposed ż and a z-plus-combining-dot index identically. That is free and it prevents the same word tokenizing two ways. Folding is a different operation with a real cost.
• Expect a 2x planning factor, not a 6x one. Polish is a case where the honest answer is undramatic. Budget context andmax_tokens at roughly twice the English figure and you will not be far wrong.
• Sorting is a separate problem with the same cause. Polish collation places ą after a and ż after z, which no byte-order sort will do; see Polish diacritics and sort order.
• Plural agreement is where token savings get spent. Polish has separate forms for one, for two-to-four, and for five and above, with the rule depending on the last two digits. A template that saves tokens by generating one form produces text that is wrong most of the time; see Polish plural forms by count range.