Why Gujarati Text Costs More Tokens Than English

Gujarati is usually introduced as Devanagari without the horizontal line across the top. That is an accurate description of how it looks and an irrelevant one for token cost, because the line is drawn by the typeface and is not encoded anywhere. What actually costs Gujarati money is that it has its own Unicode block, and so shares no bytes with Hindi even where it shares the word.

The headline stroke is not a character

Devanagari letters are drawn hanging from a continuous horizontal stroke, the shirorekha. Gujarati letters are historically the same shapes with that stroke omitted. It is a genuine and visible difference between the two scripts, and it is worth being explicit about why it changes nothing here: the stroke is part of each letter’s glyph in the font. There is no code point for it, no byte for it, and no way for a tokenizer working on UTF-8 to know whether the text it is encoding will be rendered with one.

Everything a byte-level tokenizer sees is code points, and the two scripts are structurally parallel at that level: both are abugidas, both encode vowel signs as separate code points, both use a virama to form conjuncts — U+094D in Devanagari, U+0ACD in Gujarati. The virama mechanism is explained in full on the Kannada page and applies here unchanged, so it is not repeated.

Deriving the multiplier

Gujarati : પુસ્તકાલયમાં ગુજરાતી પુસ્તકો છે.
English  : There are Gujarati books in the library.

Gujarati : 28 Gujarati code points = 28 × 3 bytes = 84
           3 spaces + full stop    =  4 × 1 byte  =  4
                                     total        = 88 bytes
           4 orthographic words

English  : 40 ASCII characters     = 40 bytes  ≈ 10 tokens

Byte-fallback ceiling: 88 tokens against about ten, so a derived worst case of 8.8x. Gujarati sits in U+0A80–U+0AFF, charted by the Unicode Consortium at U+0A80 Gujarati, which is above U+0800 and therefore three bytes per code point throughout. As everywhere in this cluster, that ceiling is arithmetic rather than a measured count, and the interesting question is where in the range a real tokenizer lands.

Same word, different bytes

Gujarati and Hindi share an enormous amount of vocabulary through Sanskrit and through centuries of contact. A speaker of one recognises a great deal of the other in writing, once the shapes are learned. A tokenizer recognises none of it, and the reason is visible in the encoding:

Gujarati પુસ્તક   ("book")
  પ  U+0AAA  →  E0 AA AA
  ુ  U+0AC1  →  E0 AB 81
  સ  U+0AB8  →  E0 AA B8
  ્  U+0ACD  →  E0 AB 8D
  ત  U+0AA4  →  E0 AA A4
  ક  U+0A95  →  E0 AA 95

Hindi पुस्तक   ("book" — same word, same phonemes)
  प  U+092A  →  E0 A4 AA
  ु  U+0941  →  E0 A5 81
  स  U+0938  →  E0 A4 B8
  ्  U+094D  →  E0 A5 8D
  त  U+0924  →  E0 A4 A4
  क  U+0915  →  E0 A4 95

Read the middle byte. Every three-byte character in this part of the plane begins E0, and in these two scripts the third byte is often identical because the blocks are laid out in the same traditional order. The middle byte is what differs, and it differs on every single character. Two spellings of the same word, phoneme for phoneme, and there is no byte sequence longer than one that they share.

Byte-pair encoding learns merges over byte sequences. Hindi is by a large margin the best-represented Indic language in web-scale corpora, so a vocabulary will hold a reasonable stock of Devanagari merges. Not one of them can fire on Gujarati text. The block boundary is a complete barrier to transfer, and it is a barrier precisely where transfer would have been most valuable, because the words on the other side of it are the same words.

The identical argument applies to Punjabi in Gurmukhi at U+0A00, and to every other Indic script that is not Devanagari. Gujarati is simply the clearest case because its lexical overlap with Hindi is so high.

It is worth noticing what this rules out. If the barrier were about the shapes — the headline stroke, the letterforms, how similar the two look on a page — then a script that looked more like Devanagari would tokenize more cheaply than one that looked less like it. Nothing of the kind happens, because the tokenizer has no access to shape. Two scripts that a reader would call visually identical and two that share no visual feature at all are in exactly the same position if they occupy different blocks, and two spellings in the same block share merges no matter how different they look. Block membership is the entire variable.

What the disjoint block costs

The first consequence is the multiplier itself. Where a Hindi document benefits from whatever Devanagari merges exist, a Gujarati document of the same content starts from a smaller stock and falls closer to byte fallback. The gap between the two is not explained by anything about Gujarati as a language; it is explained by which block the same word happens to be written in.

The second is subtler and shows up in retrieval. Because the byte overlap is zero, the subword pieces a Gujarati word is split into have no relationship to the pieces its Hindi cognate is split into. A multilingual embedding model may still place the two near each other if it learned the correspondence from parallel text, but it gets no help at all from shared surface form — which is the mechanism that makes cross-lingual retrieval work reasonably well between, say, Spanish and Italian. Splitting Indic text before embedding has its own failure modes on top of this.

The third is a budgeting rule. On the derivation above, a 4,000-token context that holds roughly 3,000 English words holds a few hundred words of Gujarati at the pessimistic end and perhaps a thousand at the optimistic one. Any chunk size, any max_tokens and any cost model derived from English behaviour needs re-deriving rather than scaling by a guess.

Measuring it on your own text

The test that matters here is the cognate test, because it makes the block barrier visible in one line of output. Encode the same word in both scripts and compare.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

pairs = [
    ("પુસ્તક", "पुस्तक"),   # book
    ("નામ",    "नाम"),      # name
    ("પાણી",   "पानी"),     # water
    ("ઘર",     "घर"),       # house
]

for guj, hin in pairs:
    g, h = enc.encode(guj), enc.encode(hin)
    print(f"gujarati {guj:8s} {len(g):2d} tokens   "
          f"hindi {hin:8s} {len(h):2d} tokens   "
          f"shared token ids: {set(g) & set(h)}")

The set of shared token ids should be empty or near-empty for every pair. That is the whole argument, printed. If the Gujarati counts are consistently higher than the Hindi ones for the same word, you are looking at the difference in training share between the two blocks rather than at any property of the Gujarati language.