Why Urdu Text Costs More Tokens Than English

Say a sentence out loud in Delhi and in Lahore and it can be the same sentence. Write it down and one version costs about a third fewer bytes than the other, because the Arabic script lives in the two-byte region of UTF-8 and Devanagari lives in the three-byte region. That is the cleanest natural experiment in this whole subject.

One language, two byte widths

Urdu is written in the Perso-Arabic script, in the Unicode Arabic block U+0600 to U+06FF plus the Arabic Supplement. Every code point in that range encodes to two UTF-8 bytes. Hindi is written in Devanagari, U+0900 to U+097F, which is three bytes per code point. Both are abugidas by behaviour in the sense that a reader supplies information the text does not fully write, but the Arabic script goes further: it is an abjad, and Urdu normally omits short vowels entirely. Fewer code points, and each one narrower.

Urdu adds letters that Arabic does not have — the retroflexes ٹ (U+0679), ڈ (U+0688) and ڑ (U+0691), the nasal ں (U+06BA), the aspiration marker ھ (U+06BE) and the barri ye ے (U+06D2). All of them are still inside the two-byte range, so they cost the same as any Arabic letter at the encoding level. They cost more at the vocabulary level, which is the subject of the fourth section below.

Digits are a trap of their own. Urdu conventionally uses the Extended Arabic-Indic digits ۰ to ۹, U+06F0 to U+06F9 in the Unicode Arabic code chart, which are a different set from the Arabic-Indic digits ٠ to ٩ at U+0660 that Arabic uses. Both sets are two bytes per digit and both are far rarer in a tokenizer’s training data than ASCII digits, so a date or an amount written in Urdu numerals can cost several times what the same number costs in ASCII. Worse, arithmetic and validation code that expects 0 to 9 will not recognise either set. Numbers are one of the few places where converting to ASCII on the way in is unambiguously correct.

Nastaliq is a typeface, not an encoding

Urdu is conventionally set in Nastaliq, the sloping cursive style in which each word cascades down and to the left, and this is worth stating plainly because it is a common source of confusion: it makes no difference to token count whatsoever. Nastaliq is a rendering decision made by a font shaping engine. The same is true of the four contextual forms every Arabic letter takes — isolated, initial, medial, final. Those forms have their own Unicode code points in the Arabic Presentation Forms blocks, but well-formed Urdu text does not use them; it stores the base letter and lets the shaper choose.

Deriving both multipliers

The sentence is “Can you call me tomorrow evening?”, in Urdu as کیا آپ کل شام مجھے فون کر سکتے ہیں؟ and in Hindi as क्या आप कल शाम मुझे फ़ोन कर सकते हैं? These are the same sentence, word for word, in the shared Hindustani vocabulary that both varieties draw on for everyday speech.

Urdu (Arabic script)
  code points ............ 35  (27 non-space)
  UTF-8 bytes ............ 62   (27 x 2, plus 8 spaces)

Hindi (Devanagari)
  code points ............ 37  (29 non-space)
  UTF-8 bytes ............ 93   (28 x 3, plus '?' plus 8 spaces)

  Urdu is 62/93 = 0.67 of Hindi's byte count for the same sentence.

English "Can you call me tomorrow evening?"
  characters ............. 33
  tokens (assumption: ~4 chars/token for English) ..... ~8

Ceilings (1 token per UTF-8 byte)
  Urdu   62 / 8 = 7.8x English
  Hindi  93 / 8 = 11.6x English

Band    (assume 2.5 bytes/token for Urdu, 3 for Hindi)
  Urdu   25 tokens -> 3.1x     Hindi  31 tokens -> 3.9x

Derived, not measured — but the 0.67 ratio between the two scripts is not derived, it is arithmetic on the UTF-8 specification, and it holds for any Urdu–Hindi pair of the same content. Check it on your own text:

import tiktoken, unicodedata
enc = tiktoken.get_encoding("o200k_base")

pairs = [
    ("ur", "کیا آپ کل شام مجھے فون کر سکتے ہیں؟"),
    ("hi", "क्या आप कल शाम मुझे फ़ोन कर सकते हैं?"),
]
for lang, s in pairs:
    s = unicodedata.normalize("NFKC", s)   # collapse presentation forms
    ids = enc.encode(s)
    print(lang, len(s), "cp", len(s.encode("utf-8")), "bytes",
          len(ids), "tokens",
          round(len(s.encode("utf-8")) / len(ids), 2), "bytes/token")

Why Arabic-script coverage does not transfer to Urdu

It is tempting to assume Urdu inherits the tokenizer coverage that Arabic enjoys, on the grounds that they share an alphabet. They do not share a vocabulary. BPE merges are learned over byte sequences, and a byte sequence is a word, not a letter. Urdu word forms — Indo-Aryan roots, Persian-derived compounds, postpositions, the auxiliary chains that carry Urdu tense — simply do not occur in Arabic text, so no merge learned from Arabic covers them. The shared alphabet buys you the single-byte-pair merges for common letter sequences and nothing above that.

This is exactly the same failure that hits Mongolian, which shares Cyrillic with Russian and inherits almost none of Russian’s efficiency; see Mongolian for the same argument on the other side of Eurasia. Persian, being closer to Urdu in loan vocabulary than Arabic is, transfers slightly better — see Persian.

Consequences for a bilingual product

• Do not average the two. If your product serves Urdu and Hindi from one budget, the same user question costs about a third more in one than the other. A per-request token cap set to the Hindi figure wastes Urdu headroom; set to the Urdu figure it truncates Hindi.
• Normalise before measuring. An Urdu corpus that came out of PDFs may be sitting in presentation forms and reading fifty percent more expensive than the same text properly encoded. NFKC first, then count.
• Right-to-left is a display problem, not a token problem. Bidirectional layout costs you nothing in tokens. It costs you in rendering, in PDF generation and in anything that concatenates Urdu with an English product name or a URL.
• Transliteration between the two is not free. Converting Urdu to Devanagari to reuse a Hindi pipeline restores the three-byte cost and introduces vowel ambiguity, because the abjad did not write the short vowels the abugida requires.