Why Telugu Text Costs More Tokens Than English

Almost every published claim about how much more a language costs is missing its denominator. Telugu is the clearest case for why that matters: depending on whether you divide by characters or by words, the same text gives multipliers that differ by a factor of two, and both are correct.

There are two ratios, not one

When somebody says a language costs 5x English, they mean one of two things and usually have not decided which.

• The per-character ratio is tokens per character of source text, divided by the same for English. It measures the script and the tokenizer, and nothing else. It is the number to use when you are deciding whether a model’s vocabulary covers your script.
• The per-meaning ratio is tokens for a passage divided by tokens for the same passage in English. It measures the script, the tokenizer, and how compactly the language writes. It is the number that predicts your bill.

These come apart whenever a language writes the same content in a different number of characters than English does — which is to say, always. Telugu and Malayalam are the instructive pair because they sit on an identical byte floor and diverge on everything above it.

What a Telugu syllable costs

The Telugu block is U+0C00 to U+0C7F, three UTF-8 bytes per codepoint, identical in width to Malayalam, Kannada and Devanagari. The word తెలుగు is six codepoints — ta, the e vowel sign, la, the u vowel sign, ga, the u vowel sign — and therefore eighteen bytes, exactly like the six-codepoint Malayalam word on the neighbouring page. On the script axis these two languages are the same language.

Where they differ is what a codepoint buys. Telugu is an abugida with a highly regular consonant-plus-vowel-sign structure and consonant clusters written with a subscript form generated by a virama sequence (U+0C4D followed by the consonant). A cluster costs three codepoints, nine bytes, for one visual stack. But Telugu writes its words with ordinary spaces and without Malayalam’s pervasive cross-word sandhi, so a Telugu whitespace token corresponds much more closely to one English word.

The result: Telugu uses more codepoints than Malayalam for the same content, because it does not compress across word boundaries, while getting better merge coverage, because its whitespace-bounded words are shorter and repeat more often. Per character, Telugu can look cheaper. Per sentence, the two land closer together than the per-character figures suggest.

The Unicode Telugu code chart documents the block range, the virama at U+0C4D, and the vowel sign repertoire.

Deriving both numbers

Assumptions, labelled. English baseline of four characters per token, per OpenAI’s published guidance. Three bytes per Telugu codepoint, from the block range. Merge efficiency m for Telugu of 1.8 to 2.4 bytes per token, on the argument above that its word-shaped units merge better than Malayalam’s. And a length assumption: Telugu writes the same content in about 0.8 codepoints per English character.

Per character. English is 0.25 tokens per character. Telugu is 3 bytes per codepoint divided by m, so at m of 2.1 that is 1.43 tokens per codepoint. The per-character ratio is 1.43 divided by 0.25, or about 5.7x.

Per meaning. A 200-character English passage is 50 tokens. The same passage in Telugu is about 160 codepoints, so 480 bytes, so at m of 2.1 about 229 tokens. The per-meaning ratio is 229 divided by 50, or about 4.6x.

Same text, same tokenizer, same assumptions, and the two answers differ by more than a fifth. Neither is wrong. They answer different questions, and a page that quotes one without saying which has told you nothing you can plan with.

Which ratio you actually need

Use the per-meaning ratio for cost forecasting, context window planning and rate limits, because those are all denominated in the work your users are trying to do rather than in characters. Use the per-character ratio when you are choosing between models, because it isolates the one thing that differs between them — the vocabulary — from everything about your text that does not change when you switch.

The distinction also explains a result that otherwise looks like a measurement error: a model can have a worse per-character ratio for Telugu and a better per-meaning ratio for your application, if its vocabulary happens to contain the specific inflected forms your domain uses. Vocabulary coverage is not uniform within a script, and the concentrated vocabulary of a narrow domain is the case where it is least uniform.

There is a third thing worth extracting from the pair, because it is free once you have both. Divide the per-meaning ratio by the per-character ratio and what remains is the length factor — how many Telugu codepoints the language spends per English character. In the derivation above that is 4.6 divided by 5.7, or about 0.8, which is the assumption the derivation started from. Computing it from measurement instead turns an assumption into an observation, and it is the one quantity on this page that belongs to Telugu alone rather than to Telugu and a tokenizer together. It will not move when a new vocabulary ships; the other two will.

That stability makes it the right thing to cache. Measure the length factor once against a good parallel corpus, and thereafter you only need the per-character ratio — which is a single pass over untranslated text — to recover the per-meaning ratio for any new model. Without the split you would need aligned translations every time you changed model, which for most teams means never doing it.

The practical numbers for Telugu, at the derived 4.6x: a 4,000-token budget holds roughly 870 English tokens’ worth of content, or about 650 English words. A 512-token retrieval chunk holds around 110 English tokens — two or three Telugu sentences. That is usually below the threshold at which a chunk is independently meaningful, so raise the chunk budget by the measured multiplier rather than keeping the English default and hoping overlap covers it.

Measuring both at once

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def both_ratios(te, en):
    t_te, t_en = len(enc.encode(te)), len(enc.encode(en))
    per_char = (t_te / len(te)) / (t_en / len(en))
    per_meaning = t_te / t_en
    return per_char, per_meaning

# aligned_pairs: a list of (telugu_sentence, english_translation) tuples
aligned_pairs = [
    ("<your Telugu sentence>", "<its English translation>"),
]

for te, en in aligned_pairs:
    pc, pm = both_ratios(te, en)
    print(f"per-char {pc:5.2f}x   per-meaning {pm:5.2f}x")

Run it over at least a few hundred aligned pairs. A single sentence gives a number with no error bar, and the variance across registers is large enough that one sentence can be off by 30 per cent in either direction. If you only have unaligned Telugu, the per-character figure is still computable and still useful; the per-meaning one is not, and estimating it from a translation you generated with the same model is circular.

For the wider Dravidian and South Indian picture, the same two-ratio split applies to Kannada and Tamil, whose scripts sit in the same three-byte range with different morphological profiles above it.