Why Arabic Text Costs More Tokens Than English

Arabic sits in a two-byte Unicode range, so the byte penalty against English is only about twice. The token penalty is usually worse than that, and the reason is almost always something optional that got into the text: vowel marks, or presentation forms from a PDF.

The bare consonantal text

Ordinary written Arabic is unvocalised. Take a labelled sentence.

هذه الميزة تستخدم ذاكرة كبيرة.
"This feature uses a large amount of memory."

Twenty-five Arabic letters, four spaces, one full stop. Arabic sits in U+0600–U+06FF, which UTF-8 encodes in two bytes, so that is 50 bytes of letters plus 5 bytes of ASCII: exactly 55 bytes. The English is 43 ASCII bytes, near 11 tokens at four characters per token.

Derive the Arabic with the assumption named. Modern tokenizers with large vocabularies contain merges for frequent Arabic letter pairs and for the definite article ال, so assume roughly one token per two or three letters for common words and one per letter for rarer ones. That puts this sentence in the region of 12 to 22 tokens against 11 for the English, so a derived multiplier of roughly 1.1× to 2×. The unconditional ceiling is the byte count, 55.

That is a mild penalty by the standards of this cluster, and it is the honest baseline: unvocalised Arabic in a modern tokenizer is not catastrophic. Everything expensive about Arabic comes from the two things below.

What diacritics cost, derived

Tashkeel — fatha, damma, kasra, sukun, shadda and the tanwin forms, U+064B to U+0652 — are separate combining code points that follow the consonant they modify. They are not part of the letter. Each is two bytes, and each is a code point the merge table almost certainly did not learn in combination with its neighbour, because the overwhelming majority of Arabic in any training corpus is unvocalised.

The arithmetic follows directly. Fully vocalised Arabic carries roughly one mark per consonant, sometimes two on a shadda-bearing letter. Add about twenty marks to the twenty-five-letter sentence above and the byte count rises from 55 to about 95 — a factor of 1.7 in bytes. But the token count rises further than the bytes do, because inserting an unfamiliar code point between two letters destroys the merge that would have joined them. A pair that was one token becomes three: letter, mark, letter. So a byte increase of 1.7× can be a token increase closer to 2.5×, and that is derived from how BPE merges work rather than measured.

Where this bites is Quranic text, classical poetry, children’s material and language-learning content, all of which are vocalised by convention. If your corpus is any of those, the multiplier you read about for “Arabic” does not apply to you and is too low.

The presentation-form trap

Arabic letters change shape depending on position in the word — initial, medial, final, isolated. In correct Unicode text this is purely a rendering matter: there is one code point per letter and the font picks the form. Unicode also defines the Arabic Presentation Forms blocks, U+FB50–FDFF and U+FE70–FEFF, which encode those shapes as distinct code points. They exist for round-tripping with legacy encodings and the Unicode Standard discourages their use in new text.

PDF text extraction produces them anyway, because the PDF stores glyphs and a naive extractor writes back the presentation-form code point that corresponds to the glyph it found. The extracted text looks correct. It tokenises much worse: presentation forms are rare in training corpora, so merge coverage is close to nil and each letter drops toward its two bytes as two separate tokens. It also fails to match ordinary Arabic in string comparison and in retrieval, which is the more damaging half.

NFKC normalisation maps presentation forms back to the standard letters. Applying it at ingestion is a one-line fix that can cut the token count of an extracted Arabic document substantially and, more importantly, makes it match the queries people type. The related failure of word order in extracted right-to-left text is a different bug with a different fix — see reordering extracted Arabic and retrieval over Arabic-script documents.

Clitics, and why word count misleads

Arabic attaches a great deal to the word. Prepositions, conjunctions, the definite article and object pronouns are written joined to the stem: وبالمكتبة is one orthographic word meaning “and in the library”, which English writes as four. This works in Arabic’s favour on a per-word basis and against it per token, because the merged form is a longer, rarer byte sequence than any of its parts, so the tokenizer breaks it up anyway — you pay for the pieces without getting the whitespace anchor that would have helped the merge table find them.

The practical implication is that comparing Arabic and English by word count is meaningless, and comparing them by character count flatters Arabic. Only the token count is comparable, which is why the script below counts tokens and not words.

Measuring and reducing it

import re
import unicodedata
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

TASHKEEL = re.compile("[" + "".join(chr(c) for c in range(0x064B, 0x0653)) + "]")

def strip_marks(s):
    return TASHKEEL.sub("", s)

ar = "هذه الميزة تستخدم ذاكرة كبيرة."

variants = (
    ("as given", ar),
    ("NFKC", unicodedata.normalize("NFKC", ar)),
    ("marks stripped", strip_marks(unicodedata.normalize("NFKC", ar))),
)

for label, s in variants:
    print(label, "codepoints", len(s),
          "bytes", len(s.encode("utf-8")),
          "tokens", len(enc.encode(s)))

Run it on a real document rather than on the example. Three outcomes are diagnostic. If the NFKC row is much smaller than the as-given row, your extractor is emitting presentation forms and you have found a bug. If the stripped row is much smaller than the NFKC row, your corpus is vocalised and you should decide deliberately whether to keep the marks. If all three are the same, the text is already clean and the multiplier you are seeing is the real cost of Arabic, which is the mild one derived above.