Why Persian Text Costs More Tokens Than English

Persian and Arabic share a script, and almost every estimate treats them as one language for token purposes. They are not. Four of the most frequent letters in Persian are code points that do not occur in Arabic, which means the merge table’s Arabic investment is largely wasted on Persian text.

Four letters that are not the Arabic ones

Persian extends the Arabic script with four letters for sounds Arabic does not have — پ, چ, ژ and گ — which is the part everybody knows. The part that matters for tokenization is subtler and far more damaging: Persian also uses different code points for two letters Arabic already has.

• Persian kaf is ک, U+06A9. Arabic kaf is ك, U+0643. Different code points, different byte sequences, near-identical appearance.
• Persian yeh is ی, U+06CC. Arabic yeh is ي, U+064A. Again different, again visually all but indistinguishable in most fonts, and the Persian form has no dots in its final and isolated shapes.

Those two letters are among the most frequent in Persian. ی in particular carries the ezafe construction, the indefinite marker and a large share of adjectival and plural morphology, so it appears in a very high proportion of Persian words. Every merge that a tokenizer learned from Arabic text and that contains ك or ي simply does not fire on correctly-typed Persian.

The consequence is that Persian gets less benefit from Arabic-script coverage than its script membership suggests, and the mild multiplier derived on the Arabic page does not transfer. The general Arabic-script arguments — two bytes per letter, diacritics as separate combining points, presentation forms from PDF extraction — all apply here too and are derived there rather than repeated.

There is a second-order problem that makes this worse in real corpora. Many Persian speakers type on keyboard layouts that emit the Arabic code points, so a substantial fraction of Persian text on the web is written with ك and ي rather than ک and ی. The same word therefore exists in two encodings, both common, with different token counts and no string equality between them. The same applies to Urdu, which extends the script further again.

The invisible character inside words

Persian orthography uses the zero-width non-joiner, U+200C, inside words. It is called نیم‌فاصله, the half-space, and it separates parts of a compound without letting the letters join cursively: می‌رود (goes), کتاب‌ها (books), خانه‌ها (houses).

For tokenization it is an unusually bad character. It is invisible, so nobody notices it in the data. It is three bytes in UTF-8, because U+200C is above U+0800 — more than the two-byte Persian letters on either side of it. And it sits in the middle of a word, so it terminates any merge that would have spanned the compound.

Worse, its use is inconsistent. The same word appears in real corpora written with a ZWNJ, with a plain space, and with nothing at all: می‌رود, می رود and میرود are all in circulation. Those are three different strings with three different token counts and three different retrieval behaviours, and only one of them is prescriptively correct. Any Persian pipeline needs a deliberate policy here, applied identically at index time and query time.

The arithmetic on a labelled sentence

این قابلیت حافظه زیادی مصرف می‌کند.
"This feature consumes a lot of memory."

Twenty-eight Persian letters at two bytes each is 56 bytes. One zero-width non-joiner adds 3. Five spaces and a full stop add 6. The sentence is exactly 65 bytes across 35 code points. The English is 38 ASCII bytes, near 10 tokens.

Derive the middle, assumption named. Assume merges exist for the most frequent Persian letter pairs and for a handful of very common words such as این and که, but that the ک and ی letters break any merge inherited from Arabic, and that the ZWNJ splits می‌کند into two unrelated fragments. That puts the sentence around 18 to 30 tokens against the English 10 — a derived multiplier of roughly 1.8× to 3×, with an unconditional ceiling of 65.

The honest caveat: that range is wider than the Arabic one, and the width is the point. Persian’s coverage depends heavily on how much Persian, as opposed to Arabic, was in the corpus that built the merge table, and that varies far more between tokenizer generations than any of the structural facts above. Measure it for your model rather than carrying the range.

Indo-European morphology in a Semitic script

It is worth saying explicitly that Persian is not a Semitic language. It is Indo-European, related to Kurdish, Pashto and more distantly to English, and it borrowed the script rather than inheriting the grammar with it. So the morphological arguments that shape Arabic’s token profile — root-and-pattern derivation, where a three-consonant root is inflected by internal vowel changes — do not apply.

Persian instead uses prefixes, suffixes and analytic constructions. Verbs take a prefix for aspect (می-) and person endings; the ezafe particle links nouns to modifiers and is written as a bare ی or left unwritten entirely; plurals are formed with -ها or -ان. These are concatenative and regular, which is good news for a merge table in principle: the same suffix is the same byte sequence every time, unlike in Turkish, where vowel harmony multiplies each morpheme into several forms.

The bad news is that the two most common of those markers — the imperfective می- and the plural -ها — are exactly the ones normally written with a ZWNJ. Persian’s morphology is friendly to BPE and its orthography puts a merge-breaking invisible character in the middle of the friendliest parts.

Measuring it, and what to normalise

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

ARABIC_TO_PERSIAN = ((chr(0x0643), chr(0x06A9)),
                     (chr(0x064A), chr(0x06CC)))

def to_persian(s):
    for a, p in ARABIC_TO_PERSIAN:
        s = s.replace(a, p)
    return s

fa = "این قابلیت حافظه زیادی مصرف می‌کند."

rows = (
    ("as given", fa),
    ("normalised letters", to_persian(fa)),
    ("zwnj removed", to_persian(fa).replace(chr(0x200C), "")),
    ("zwnj to space", to_persian(fa).replace(chr(0x200C), " ")),
)

for label, s in rows:
    print(label,
          "codepoints", len(s),
          "bytes", len(s.encode("utf-8")),
          "tokens", len(enc.encode(s)))

Three things to read from that. If the first two rows differ, your corpus contains Arabic code points in Persian words and you should normalise them — the direction is Arabic to Persian, not the reverse, because Persian orthography is the correct target for Persian text. If the ZWNJ rows differ substantially in tokens, the half-space is a real cost in your data and the policy question is live. And whichever variant you pick, apply it to queries and documents identically, or retrieval will fail on exactly the common words this page is about.

• Normalise once, at ingestion. Letters, digits and ZWNJ policy together, before token counting or embedding.
• Do not strip the ZWNJ blindly. It is semantically meaningful in some compounds, and removing it can merge two words into a different one.
• Punctuation is mirrored too. Persian uses ، for comma and ؟ for question mark, distinct code points from their ASCII counterparts — see Persian punctuation in right-to-left text.