Why Punjabi Text Costs More Tokens Than English

Punjabi has more first-language speakers than Italian, German and Korean combined, and it tokenizes like a language nobody has heard of. The reason is not the number of speakers. It is that those speakers write in two different scripts, and a byte-level tokenizer learns each one from only its own half of the text.

The byte floor for Gurmukhi

Every production tokenizer in wide use — OpenAI’s cl100k_base and o200k_base, the SentencePiece vocabularies shipped with the Llama and Mistral families — is byte-level BPE. It does not see characters. It sees the UTF-8 bytes of your string and merges adjacent byte pairs according to a merge table learned once, from a training corpus, before the model existed.

That makes the first number on this page a property of Unicode rather than of any vendor. The Gurmukhi block occupies U+0A00 to U+0A7F, which puts every one of its codepoints in the three-byte range of UTF-8. Take the word ਪੰਜਾਬੀ. It is six codepoints — the consonant pa, a tippi for nasalisation, ja, the aa vowel sign, ba, and the ii vowel sign — and therefore eighteen UTF-8 bytes. The English word “Punjabi” is seven ASCII bytes and, in cl100k_base, two tokens.

Eighteen bytes is the ceiling on how badly this can go: if the merge table contains nothing for Gurmukhi at all, the word is eighteen tokens, because a byte-level tokenizer always has a fallback and the fallback is one token per byte. Every merge the vocabulary happens to contain pulls the real number down from eighteen. So the whole question — the entire difference between a 3x page and a 9x page — is how many Gurmukhi merges got learned.

The Unicode Consortium publishes the Gurmukhi code chart, which is where the block range and every character name used on this page comes from.

Deriving the multiplier

Here is the arithmetic, with every assumption named so you can disagree with a specific one rather than with the result.

• Assumption 1, the English baseline. OpenAI’s help documentation gives a rule of thumb of roughly four characters per token for ordinary English prose. Call a 200-character English sentence 50 tokens.
• Assumption 2, script width. Three UTF-8 bytes per Gurmukhi codepoint. This one is not an assumption at all; it follows from the block range.
• Assumption 3, merge efficiency. Call m the average number of bytes a Gurmukhi merge consumes. For a script the vocabulary has never seen, m is 1. For a very well-merged script like English, m is around 4. For Gurmukhi, a plausible range is 1.5 to 2.5 — some merges exist, most of the common combining sequences do not.
• Assumption 4, comparable length. Punjabi writes the same meaning in fewer codepoints than English writes in characters, because vowels are diacritics rather than full letters. Call it 0.7 Gurmukhi codepoints per English character for the same content.

Put them together for a sentence whose English is 200 characters. The Punjabi is about 140 codepoints, so 420 bytes, so 420 divided by m tokens. At m of 2 that is 210 tokens against English’s 50: a multiplier of about 4.2x. At m of 1.5 it is 5.6x, and at the byte floor with no merges at all it is 8.4x.

One language, two scripts, half a corpus

Now the part that is specific to Punjabi and to almost nothing else at this scale. Punjabi is written in Gurmukhi in Indian Punjab and in Shahmukhi, a Perso-Arabic script, in Pakistani Punjab, where the larger share of speakers lives. These are not two fonts for one encoding. They are disjoint Unicode blocks with no shared bytes whatsoever: Gurmukhi at U+0A00, Arabic at U+0600.

BPE allocates its merge budget by frequency over the whole training corpus. A merge earns a slot in the vocabulary by appearing often enough, competing against every other candidate sequence in every other language. Splitting a language’s text across two encodings means every frequent Punjabi word competes for two slots instead of one, each at roughly half the frequency it would have had. A word that would have just cleared the threshold as a single script clears it in neither.

This is why speaker count is a poor predictor of token cost and web corpus share is a good one. The comparison to make is with Hindi, which has one script, a much larger digital news and Wikipedia presence, and consequently a meaningfully better ratio despite sitting in the same three-byte range. The Shahmukhi side inherits a different set of problems, closer to those described for Urdu, whose script it shares.

The marks English does not have to spell

Gurmukhi spells out, as separate codepoints, several things English encodes inside a letter it already has a token for.

• Addak (U+0A71) marks gemination. Where English writes a doubled consonant and pays nothing extra because the doubled form is already a merge, Gurmukhi adds a whole codepoint — three more bytes.
• Tippi (U+0A70) and bindi (U+0A02) mark nasalisation and are chosen by context. Two codepoints for one phonological feature, each diluting the other’s frequency.
• Subscript consonants — the pairin forms of ha, ra and va — are encoded as a virama (U+0A4D) followed by the consonant. That is a three-codepoint, nine-byte sequence for what is visually one stacked glyph.

Each of these is a place where the codepoint count runs ahead of the glyph count, and the tokenizer bills the codepoints. It is also why a Punjabi string can look shorter than its English translation on screen and cost four times as much.

Measure it on your own text

Do not take the derivation. Run this against a few hundred sentences of the Punjabi you actually send — product copy tokenizes differently from Gurbani, and both differ from user-generated chat.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

pairs = [
    ("<your Gurmukhi sentence>", "<its English translation>"),
]

for pa, en in pairs:
    t_pa = len(enc.encode(pa))
    t_en = len(enc.encode(en))
    b_pa = len(pa.encode("utf-8"))
    # m = bytes consumed per token: 1.0 means no merges fired at all
    print(f"{t_pa:4d} tok  {t_pa / t_en:5.2f}x  m={b_pa / t_pa:4.2f}")

The m column is the one to watch. If it sits near 1.0 your text is being tokenized byte by byte and no amount of prompt tuning will help — the fix is a model whose tokenizer saw Gurmukhi, or transliteration, for which see Gurmukhi to Latin transliteration. If m is above 2 the vocabulary has real Punjabi coverage and your remaining cost is the three-byte floor, which nothing can remove.

The practical consequence: at 4.2x, a 4,000-token context window holds roughly the Punjabi equivalent of 950 English tokens, or about 700 English words. A retrieval chunk sized at 512 tokens holds two or three Punjabi sentences rather than a paragraph, which is usually too little context for the chunk to be independently meaningful. Size chunks in characters of source text and convert, rather than reusing an English-derived token number.