Why Tamil Text Costs More Tokens Than English

Tamil is expensive for two independent reasons, and reporting them as one multiplier hides the fact that only one of them can be reduced. The encoding cost is fixed. The morphology cost depends on how you write the sentence, and it is the larger of the two.

A large Brahmic character set at three bytes each

Tamil occupies U+0B80 to U+0BFF, inside the three-byte region of UTF-8. Like the other Brahmic scripts it is an abugida: twelve vowels and eighteen consonants combine into the 247 conventional letterforms of the script, and most of those combinations are written as a consonant code point followed by a dependent vowel sign code point. The pulli, U+0BCD, is the Tamil virama, and it strips the inherent vowel to produce a bare consonant.

Tamil differs from Bengali and Devanagari in one respect that helps slightly: its orthography largely avoids stacked conjuncts, so a consonant cluster is written as consonant + pulli + consonant in linear order rather than as a fused ligature. Fewer of the code points are invisible control characters. The encoding is still three bytes per code point, and two or three code points per syllable, so the floor is high regardless.

There is a second tier of Tamil characters that behaves worse than the core alphabet. The Grantha letters ஜ, ஷ, ஸ and ஹ, listed in the Unicode Tamil code chart, exist to write Sanskrit and foreign loanwords, and Tamil brand names, transliterated English and technical vocabulary lean on them heavily. They are individually much rarer in any corpus than the native consonants, so they are the least likely to have merges. The Tamil digits U+0BE6 to U+0BEF and the year, month and day signs are rarer still. If your text is product names, addresses or invoice fields rather than running prose, expect it to sit closer to the ceiling than the derivation below suggests.

One word, five English words’ worth of grammar

Tamil is strongly agglutinative. Tense, aspect, person, number and case attach as ordered suffixes to a root, and the result is a single orthographic word with no spaces in it. The verb form படித்துக்கொண்டிருக்கிறேன் means “I am reading” and decomposes into a reading root, a perfective auxiliary, a continuous auxiliary, a present-tense marker and a first-person-singular ending. English writes that as three separate words, two of which are among the highest-frequency tokens in any vocabulary.

This is the part that hurts a BPE tokenizer far more than the byte width does. BPE learns merges for byte sequences that recur often. In an agglutinative language the number of distinct surface forms of a single root is enormous, so each form individually is rare, so no merge covers it, so it fragments. English gets cheap tokens precisely because it spreads its grammar across separate high-frequency function words that appear identically in millions of documents. Tamil concentrates that grammar inside word forms that each appear comparatively rarely. The same mechanism drives the cost of Turkish and Finnish, which pay it without any byte-width penalty at all.

Deriving the multiplier

The sentence is நான் புத்தகத்தைப் படித்துக்கொண்டிருக்கிறேன். and its gloss is “I am reading the book.” It is deliberately verb-heavy: it is chosen to show the morphology cost, and a noun-heavy Tamil sentence will land lower than this one.

Tamil sentence
  code points ............ 44  (42 non-space)
  UTF-8 bytes ............ 126  (41 Tamil code points x 3, plus 3 ASCII)
  orthographic words ..... 3

English gloss "I am reading the book."
  characters ............. 22
  tokens (assumption: ~4 chars/token for English) ..... ~6

Ceiling  (1 token per UTF-8 byte, worst case)
  126 / 6 = 21x English

Band     (assume 3 bytes/token, ~1 token per code point)
  126 / 3 = 42 tokens  ->  42 / 6 = 7.0x English

Band     (assume 4.5 bytes/token, a vocabulary with real Tamil merges)
  126 / 4.5 = 28 tokens  ->  28 / 6 = 4.7x English

Derived, not measured. The high ceiling here is partly an artefact of how short the English gloss is — five very common English words tokenize almost for free. That is not a distortion, it is the point: the cheapest possible English against a form Tamil considers unremarkable.

Why the two costs multiply

If Tamil were written in Latin script, the morphology cost would remain and the multiplier would still be well above one. If Tamil were isolating but written in its own script, the byte cost would remain. Tamil has both, and they compose rather than add: rare word forms mean fewer merges apply, and every unmerged fragment is priced at three bytes per code point rather than one. The measurement worth making is the split, and this script makes it:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

# The same meaning, agglutinated and then broken apart with spaces at
# morpheme boundaries. Any difference is the morphology cost alone --
# the byte count is nearly identical either way.
packed  = "படித்துக்கொண்டிருக்கிறேன்"
split   = "படித்து க் கொண்டு இருக்கிறேன்"

for label, s in (("packed", packed), ("split", split)):
    ids = enc.encode(s)
    print(label, len(s.encode("utf-8")), "bytes",
          len(ids), "tokens",
          round(len(s.encode("utf-8")) / len(ids), 2), "bytes/token")

If the split form costs meaningfully fewer tokens per byte, the vocabulary has merges for the individual morphemes but not for the combined surface form, and your Tamil cost is dominated by morphology. If both are similar, the vocabulary is operating near byte fallback and the cost is dominated by the encoding.

What to do about it

• Budget in tokens, per language, at the point of chunking. At the middle of the band above, a 4,000-token context holds roughly 4,000 Tamil code points, which is on the order of 300 to 400 Tamil words. The same window is several thousand English words.
• Do not romanize to save tokens. It works — Latin Tamil is one byte per character and merges well — and it costs you retrieval quality, because the embedding model was trained on Tamil script and a transliterated query will not match a Tamil-script document.
• Expect the output side to be worse than the input side. Providers price output tokens several times higher than input, and Tamil generation pays the multiplier on the expensive side of that ratio. A Tamil chatbot answer of the same length as an English one is a materially different line on the bill.
• Check truncation, not just cost. A max_tokens value tuned against English answers cuts Tamil answers off in the middle of a word form, which in an agglutinative language can invert the meaning by dropping a negation suffix.
• Index on the root, not the surface form. Keyword search over Tamil fails for the same reason the tokenizer struggles: the form in the document and the form in the query differ by a chain of suffixes. A morphological analyser at index time is worth more than any amount of tuning at query time.

Malayalam shares Tamil’s three-byte Brahmic encoding and adds more conjunct ligatures, so it sits slightly worse on the same axis; see Malayalam for that variant rather than assuming the Tamil number transfers.