Why Malayalam Text Costs More Tokens Than English

Malayalam is expensive for a reason that is not shared with the other Dravidian scripts: its orthography glues words together. A byte-level BPE tokenizer learns most of its useful merges inside whitespace-bounded words, and Malayalam gives it far fewer whitespace boundaries to work with.

Start from one word

Take the name of the language itself, മലയാളം. That is six codepoints: ma, la, ya, the aa vowel sign, La (the retroflex lateral), and an anusvara. The Malayalam block runs from U+0D00 to U+0D7F, entirely inside UTF-8’s three-byte range, so those six codepoints are eighteen bytes. The English word “Malayalam” is nine ASCII bytes.

Six visible units, eighteen bytes, and a tokenizer that has to find merges in them. If the vocabulary contains the whole word, it is one token. If it contains nothing, it is eighteen. That range — an eighteen-fold spread on one common word — is the whole subject.

The Unicode Malayalam code chart gives the block range and the character names used here.

Sandhi removes the boundaries BPE needs

BPE learns merges from adjacent byte pairs, and in practice the merges that survive into a vocabulary are overwhelmingly whole frequent words and word-initial fragments, usually including the leading space. That is why the token for “ the” with a space is different from “the” without one, and why word frequency in the training corpus is what decides whether a word gets its own token.

Malayalam orthography applies sandhi across word boundaries and writes the result closed. Postpositions, case markers, auxiliaries, negation and conjunctions attach to the preceding word, and the junction is often spelled with a phonological change rather than a seam. The consequence is that a Malayalam “word”, in the sense of a whitespace-delimited string, frequently corresponds to what English writes as three or four words.

For the tokenizer this is doubly bad. The long joined form is rare, so it earns no merge of its own. And it cannot be assembled from merges for its parts, because the parts do not appear at a boundary the merge table recognises — the sandhi change means the stem-final bytes are not the bytes the stem’s own merge ends with. So the long form falls back much further toward the byte floor than its length would suggest.

This is the mechanism that separates Malayalam from Tamil and Telugu, which share the three-byte floor exactly but write shorter orthographic words. The floor is the same; the distance above it is not.

Two orthographies, one language

Malayalam has a second, quieter cost. The script reform of the early 1970s introduced a simplified set of forms for several consonant-vowel combinations and conjuncts, and the older forms did not go away. Modern Unicode text contains both: conjuncts written as an explicit virama sequence with a zero-width joiner to request the stacked form, and the same conjuncts written in the reformed spelling.

Two encodings of the same word means each appears at roughly half the frequency, and BPE’s merge selection is a frequency threshold. A word on the edge of earning a merge earns none in either spelling. Worse for anything downstream, the two encodings are not equal under naive string comparison, so a retrieval index built on one spelling misses queries written in the other. That is a normalisation problem before it is a tokenizer problem, and it is worth handling first.

The derivation

Assumptions, all four stated so the result can be argued with:

• English baseline of about four characters per token, per OpenAI’s published rule of thumb. A 240-character English passage is about 60 tokens.
• Three UTF-8 bytes per Malayalam codepoint, which follows from the block range and is not negotiable.
• Malayalam writes the same content in roughly 0.65 codepoints per English character — fewer units, because vowels are signs and because sandhi removes spaces. So about 156 codepoints, or 468 bytes.
• Merge efficiency m, bytes consumed per token. The sandhi argument above predicts Malayalam sits at the low end for a script of its corpus size: call it 1.4 to 2.0.

468 bytes at m of 1.7 is about 275 tokens, against 60 for the English: a multiplier near 4.6x. At m of 1.4 it is 5.6x. At the byte floor, 7.8x. Note that the fewer-codepoints assumption is working in Malayalam’s favour here and the sandhi assumption against it, and they partly cancel — which is exactly why quoting a single number for a language is a mistake without saying which assumption is doing the work.

What this does to a RAG chunk

At roughly 4.6x, a chunk size of 512 tokens holds about 110 English tokens’ worth of Malayalam — somewhere around 80 English words of content. A chunking strategy tuned on English documents and reused unchanged will produce Malayalam chunks that are too small to answer anything, and a retriever that returns five of them still has less context than one English chunk.

Two corrections, in order of how much they help. First, chunk on sentence boundaries in the source text and let the token count fall where it falls, rather than fixing a token budget. Malayalam sentence boundaries are marked with an ordinary full stop and are easy to find. Second, raise the token budget by the measured multiplier for your own corpus, not by a number from a page like this one.

import tiktoken, unicodedata

enc = tiktoken.get_encoding("o200k_base")

def profile(text):
    nfc = unicodedata.normalize("NFC", text)
    toks = enc.encode(nfc)
    zwj = nfc.count("\u200d")
    return {
        "codepoints": len(nfc),
        "bytes": len(nfc.encode("utf-8")),
        "tokens": len(toks),
        "bytes_per_token": len(nfc.encode("utf-8")) / len(toks),
        "zwj_count": zwj,
    }

print(profile(open("corpus.ml.txt", encoding="utf-8").read()))

If zwj_count comes back high, normalising the two orthographies to one before indexing will buy you more than any tokenizer change. And if bytes_per_token is close to 1.0, the model you are calling has essentially no Malayalam in its vocabulary, which usually shows up as poor generation quality as well as high cost.