Why Nepali Text Costs More Tokens Than English

Nepali and Hindi are written in the same script, so any argument that explains their token cost purely in terms of Devanagari has to predict they cost the same. They do not, and the gap between them is the clearest available measurement of how much of a language’s token cost is the script and how much is the language.

The script is identical, the language is not

Devanagari occupies U+0900 to U+097F, three UTF-8 bytes per codepoint. The word नेपाली is six codepoints — na, the e vowel sign, pa, the aa vowel sign, la, the ii vowel sign — and therefore eighteen bytes. A Hindi word of the same length costs the same eighteen bytes. On the script axis, defined on the Indonesian page as bytes per character, Nepali and Hindi are indistinguishable.

They are not indistinguishable on the merge axis, and that axis is language-specific even when the script is not. A merge is a byte sequence. A byte sequence corresponds to a spelling. Two languages sharing a script share merges exactly to the extent that they share spellings — which for Hindi and Nepali is a great deal, and not everything.

The Unicode Devanagari code chart is shared by both languages, along with Marathi, Sanskrit and several others, which is why the script argument alone cannot distinguish any of them.

Where Nepali free-rides on Hindi

Hindi has a far larger web presence than Nepali, so essentially all the Devanagari merges in any general-purpose vocabulary were bought with Hindi text. Nepali gets to use them wherever it happens to spell things the same way, and the overlap is substantial:

• Sanskrit-derived vocabulary. Both languages draw heavily on tatsama forms — words taken from Sanskrit with unchanged spelling — across formal, legal, administrative and technical registers. These are byte-identical and merge identically.
• The digits and the danda. The Devanagari digits and the sentence-final danda (U+0964) are shared, frequent, and certainly merged.
• Conjunct machinery. The virama at U+094D and the common conjunct sequences built with it appear in both, so whatever merges exist for frequent conjuncts fire for Nepali too.
• Loanwords. International vocabulary transliterated into Devanagari tends to be spelled similarly in both.

This free-riding is why Nepali is not as expensive as its own corpus share would predict. A language with Nepali’s web presence in a script nobody else used would sit close to the byte floor. Sharing Devanagari with Hindi is worth a great deal.

Where it stops free-riding

The free ride ends at the parts of Nepali that are spelled differently, and those parts are unfortunately the most frequent parts of any text.

• Verb morphology. Nepali’s copula and auxiliary system — छ, छन्, थियो and their agreement forms — is not Hindi’s, and these are among the highest-frequency strings in the language. High frequency within Nepali, low frequency in the corpus, no borrowed merge.
• Postpositions and case markers. The two languages use overlapping but non-identical sets, and the ones that differ appear in nearly every sentence.
• Honorific levels. Nepali grammaticalises several degrees of honorification in verb endings, multiplying the number of surface forms for the same verb in a way Hindi does not match.
• Core native vocabulary. The everyday non-Sanskritic lexicon diverges substantially, and everyday vocabulary is by definition what most text is made of.

So the structure of the problem is: Nepali gets Hindi rates on its formal and borrowed vocabulary and near-floor rates on its grammatical core. That is a weighted average, and it is derivable.

Deriving the multiplier as a weighted average

Assumptions, all labelled. English at four characters per token, per OpenAI’s published rule of thumb, so 0.25 tokens per character. Three bytes per Devanagari codepoint, from the block range. A Hindi merge efficiency of 2.4 bytes per token — the reference figure, itself an assumption, and the one to replace first with a measurement. A near-floor efficiency of 1.2 bytes per token for unshared Nepali material. And a shared fraction: call it 60 per cent of running text riding on Hindi merges, 40 per cent not.

The blended efficiency is not the arithmetic mean of the two efficiencies, because efficiency sits in the denominator. Work in tokens per byte instead: 0.417 for the shared part, 0.833 for the unshared part. Weighted, that is 0.6 times 0.417 plus 0.4 times 0.833, which is 0.583 tokens per byte, or a blended m of about 1.72 bytes per token.

Three bytes per codepoint at m of 1.72 is 1.74 tokens per codepoint. Assume Nepali writes the same content in 0.75 codepoints per English character and a 200-character English passage is 150 Nepali codepoints, so about 261 tokens against English’s 50 — a per-meaning multiplier of about 5.2x.

Run the same arithmetic with pure Hindi efficiency throughout and you get about 3.75x. The difference between those two figures — roughly 1.4x — is the part of Nepali’s cost that is the language rather than the script, and it is the number that a script-only analysis makes invisible. Compare against the Hindi derivation before quoting either.

Estimating your own shared fraction

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

ne = open("corpus.ne.txt", encoding="utf-8").read()

# words whose bytes-per-token is high are riding on borrowed merges;
# words near 1.0 are falling back to the byte floor
shared, unshared = 0, 0
for w in ne.split():
    b = len((" " + w).encode("utf-8"))
    t = len(enc.encode(" " + w))
    if b / t >= 2.0:
        shared += 1
    else:
        unshared += 1

print(f"well-merged {shared}  near-floor {unshared}  "
      f"shared_fraction={shared / (shared + unshared):.2f}")

Feed that fraction back into the weighted average above and you have a multiplier derived from your own text rather than from a plausible guess. If it comes back well under 0.5, transliteration is worth evaluating — Devanagari to Latin transliteration discusses the trade-off, which is real, because transliterating cuts the token count and usually costs generation quality.