Why Kannada Text Costs More Tokens Than English

Kannada sits in the Unicode block U+0C80–U+0CFF, so every character takes three bytes in UTF-8. That alone would give a multiplier around three. The observed multipliers for Indic scripts are much higher than that, and the reason is that a Kannada word contains more code points than it contains shapes.

The characters you cannot see

Kannada is an abugida. A consonant letter carries an inherent vowel; other vowels are written as marks attached to the consonant; and a consonant with no vowel at all is written by attaching a suppressor. Unicode encodes all three of those as separate code points in the logical order they are pronounced, and the rendering engine assembles them into the clusters you actually see.

The expensive one is the vowel suppressor, called the virama and encoded at U+0CCD. It has no shape of its own. Its job is to say “this consonant has no vowel, join it to the next one”, which the font renders by moving the following consonant into a subscript form beneath the first — the ottakshara. So a conjunct that occupies one visual position on the page costs three code points and nine bytes: consonant, virama, consonant.

Every vowel sign is likewise its own code point. The result is a systematic gap between what a reader counts and what a tokenizer counts, and it runs in the direction that surprises people:

ಗ್ರಂಥಾಲಯದಲ್ಲಿ   ("in the library")

code points : ಗ ್ ರ ಂ ಥ ಾ ಲ ಯ ದ ಲ ್ ಲ ಿ      = 13
visual clusters : ಗ್ರಂ  ಥಾ  ಲ  ಯ  ದ  ಲ್ಲಿ    =  6
UTF-8 bytes : 13 × 3                          = 39

Six shapes, thirteen code points, thirty-nine bytes. Two of those thirteen are viramas that a reader never sees as characters at all, and under byte fallback each of them costs three tokens.

Deriving the multiplier

Kannada : ಗ್ರಂಥಾಲಯದಲ್ಲಿ ಕನ್ನಡ ಪುಸ್ತಕಗಳು ಇವೆ.
English : There are Kannada books in the library.

Kannada : 30 Kannada code points  = 30 × 3 bytes = 90
          3 spaces + full stop    =  4 × 1 byte  =  4
                                    total        = 94 bytes
          4 orthographic words

English : 39 ASCII characters     = 39 bytes  ≈ 10 tokens
          7 orthographic words

The byte-fallback ceiling is 94 tokens against roughly ten for English: a derived worst case of 9.4x. The floor, if all four words were single learned tokens, is under one. Neither is a measurement; both are arithmetic from the byte count, using the roughly four characters per token that OpenAI publishes for English on its tokenizer page.

Where does the real number sit? Higher up the range than for two-byte scripts, and for a structural reason rather than a statistical one. Byte-pair merges are learned on frequent byte sequences. In Kannada, the most frequent sequences are consonant plus vowel sign and consonant plus virama plus consonant — that is, the sequences that make up a cluster. A vocabulary with even modest Kannada exposure will learn some of those, so the common clusters merge and the rest do not. That is why Indic scripts show wide variance between documents: technical or literary vocabulary with unusual conjuncts falls much closer to the ceiling than everyday prose does.

Two published studies quantify the general phenomenon across many languages: Petrov and colleagues in “Language Model Tokenizers Introduce Unfairness Between Languages” (2023) and Ahia and colleagues in “Do All Languages Cost the Same?” (2023). Both report that the worst-served scripts cost many times what English does for identical content, and both put Indic scripts near the expensive end.

Kannada and Telugu are the same arithmetic

Kannada and Telugu descend from the same Old Kannada script and remain structurally parallel: the same abugida logic, the same vowel-sign system, the same virama mechanism, and blocks laid out in the same order. Kannada occupies U+0C80–U+0CFF and Telugu U+0C00–U+0C7F, adjacent and charted by the Unicode Consortium at U+0C80 Kannada and U+0C00 Telugu.

The correspondence is close enough that the same word written in the two scripts usually has the same number of code points, and therefore exactly the same byte count. There is no script-level cost difference between them to find. Anything that separates their real token counts is training share — how much text in each language survived corpus construction — and not the writing system. This is worth stating plainly because the two are often compared as though the glyphs mattered. The Telugu page carries the training-share argument rather than repeating this one.

The same holds broadly for Malayalam and Tamil, with one real exception: Tamil uses far fewer conjunct clusters in its modern orthography, so it emits fewer viramas per word and lands slightly cheaper than the other three for reasons that are genuinely about the script.

What this does to a retrieval pipeline

The multiplier is not merely a bill. It changes the behaviour of any system that was sized in characters.

• Chunking by character count silently overflows. A 1,000-character chunk is about 250 tokens of English and, on the derivation above, up to 3,000 tokens of Kannada. If your embedding model truncates at 512 tokens, most of every Kannada chunk is discarded without an error — the vector is computed from the beginning of the chunk and nothing tells you the rest was dropped.
• Splitting in the wrong place corrupts the text. A splitter that cuts at a fixed character offset can land between a consonant and its vowel sign, or between a consonant and its virama. The result is not a truncated word, it is an ill-formed cluster that renders as a dotted circle and embeds as noise.
• Top-k retrieval returns less context than you think. Five retrieved chunks that fit comfortably in an English prompt can exceed the window in Kannada, and the failure arrives as a truncated prompt rather than as a rejected request.

Measuring it on your own text

The specific claim to test is that viramas are expensive. Compare words with and without conjuncts and look at the cost per visible cluster rather than per character.

import tiktoken, unicodedata, regex

enc = tiktoken.get_encoding("o200k_base")

def clusters(s):
    # grapheme clusters ≈ what a reader counts as one shape
    return regex.findall(r"\X", s)

for word in ["ಪುಸ್ತಕ", "ಕನ್ನಡ", "ಗ್ರಂಥಾಲಯ", "ಮನೆ"]:
    cps = len(word)
    gs  = len(clusters(word))
    vir = word.count("\u0CCD")
    tok = len(enc.encode(word))
    print(f"{word:12s} {gs:2d} shapes  {cps:2d} code points  "
          f"{vir} viramas  {len(word.encode('utf-8')):3d} bytes  {tok:3d} tokens")

If tokens track code points rather than shapes — and in particular if the words with viramas cost disproportionately more per shape — the mechanism on this page is what you are paying for. The regex module is needed for \X; Python’s built-in re does not support grapheme clusters.