Why Bengali Text Costs More Tokens Than English

Bengali bills roughly five to fifteen times what the same sentence costs in English, and the two halves of that penalty have different causes and different fixes. One half is fixed by the Unicode encoding and will never improve. The other half is a property of the tokenizer vocabulary and improves every time a vendor ships a new one.

Three bytes per code point, before anything else

The Bengali–Assamese script occupies U+0980 to U+09FF in Unicode. Everything in that range sits in the three-byte region of UTF-8, so a single Bengali letter costs three bytes on the wire where a Latin letter costs one. Byte-level BPE, the family of tokenizer used by GPT-2 and by every OpenAI vocabulary since, operates on those UTF-8 bytes rather than on characters. That design choice is why nothing is ever out-of-vocabulary: any byte sequence can be represented, in the worst case one token per byte. The price of never failing is that unfamiliar text degrades smoothly toward that worst case instead of erroring.

So the ceiling is exact and it is not an estimate. A Bengali string of n UTF-8 bytes can never take more than n tokens, and it takes exactly n only if the vocabulary contains no merge covering any part of it. Bengali is nowhere near that bad in a modern vocabulary, but it is much closer to it than English, which is typically five to six bytes deep into merged word-pieces.

One visible cluster, several code points

Bengali is an abugida, not an alphabet. A consonant letter carries an inherent vowel, and any other vowel is written as a dependent sign that is its own code point. Then there is the hasanta, U+09CD, the virama that suppresses the inherent vowel and binds two consonants into a conjunct. It is a control character. It costs three bytes and it renders as nothing at all — its entire effect is on the shape of the letters around it.

Take one word from the sentence used below. The genitive বিশ্ববিদ্যালয়ের (“of the university”) is, to a reader, about seven visual clusters. To the tokenizer it is sixteen code points — ব, ি, শ, ্, ব, ব, ি, দ, ্, য, া, ল, য, ়, ে, র — and therefore forty-eight UTF-8 bytes. English writes “university’s” in twelve. That is a four-to-one byte ratio on one word before a single question about training data has been asked, and it is the reason a Bengali page is expensive even on a model whose tokenizer handles Bengali reasonably well.

The same abugida mechanics apply to Devanagari, which is why Hindi lands in a similar band. The difference between the two is corpus share, not structure.

Deriving the multiplier

Here is the arithmetic, with every assumption named. The Bengali sentence is বিশ্ববিদ্যালয়ের শিক্ষার্থীরা পরীক্ষার জন্য প্রস্তুতি নিচ্ছে। and its English gloss is “University students are preparing for the exam.”

Bengali sentence
  code points ............ 61  (56 non-space)
  UTF-8 bytes ............ 173  (56 x 3, plus 5 ASCII spaces)

English gloss
  characters ............. 47
  tokens (assumption: ~4 chars/token, OpenAI's published
          rule of thumb for English) ......... ~12

Ceiling  (byte-level BPE cannot exceed 1 token per byte)
  173 tokens / 12 = 14.4x English

Band     (assume the vocabulary resolves Bengali at
          3 bytes/token, i.e. roughly one token per code point)
  173 / 3 = 58 tokens  ->  58 / 12 = 4.8x English

Band     (assume a better-covered vocabulary at 4 bytes/token)
  173 / 4 = 43 tokens  ->  43 / 12 = 3.6x English

Every number above is derived, not measured. The honest statement is a range: this sentence is somewhere between about 3.5x and 14.4x its English gloss, and where it falls inside that range is entirely a question of which merges the vocabulary happens to contain. Run this on your own corpus rather than trusting the band:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
s = "বিশ্ববিদ্যালয়ের শিক্ষার্থীরা পরীক্ষার জন্য প্রস্তুতি নিচ্ছে।"
ids = enc.encode(s)

print("code points ", len(s))
print("utf-8 bytes ", len(s.encode("utf-8")))
print("tokens      ", len(ids))
print("bytes/token ", round(len(s.encode("utf-8")) / len(ids), 2))

# The diagnostic that matters: how much of this fell back to raw bytes?
lone = sum(1 for i in ids if len(enc.decode_single_token_bytes(i)) == 1)
print("single-byte tokens", lone, "of", len(ids))

The last line is the one to watch. A high single-byte count means the vocabulary has no Bengali merges to offer and is spelling your text out one UTF-8 byte at a time. A low count means Bengali word-pieces exist and the remaining cost is the encoding, which nothing can fix.

Speakers are not corpus share

Bengali is among the most spoken languages in the world by native speakers, on the order of a quarter of a billion people across Bangladesh and eastern India. It is not among the most written languages on the crawled web, and the tokenizer is trained on the latter — a mismatch that shows up in model quality as well as in cost, and is treated separately under why Bengali support lags its speaker population. Common Crawl publishes per-language document counts for each of its crawls; Bengali sits at a fraction of one percent against English above forty percent, and a BPE vocabulary allocates merges in proportion to what it saw.

Petrov and colleagues at Oxford quantified the consequence across languages in “Language Model Tokenizers Introduce Unfairness Between Languages”, published at NeurIPS 2023, reporting differences of up to roughly fifteen times in token count for the same content. That paper is the right citation for the general shape; the number for your text and your model is the script above.

What a 4,000-token budget holds in Bengali

Assume the middle of the band, three bytes per token. A 4,000-token window then holds about 12,000 UTF-8 bytes, which is about 4,000 Bengali code points. The sentence above averages a little over nine code points per word, so 4,000 tokens is roughly 430 Bengali words. The same 4,000 tokens in English, at four characters per token, is about 16,000 characters and somewhere near 2,700 words.

• RAG chunks sized in characters are wrong by six times. A 1,000-character chunk rule tuned on English produces Bengali chunks that overflow the embedding model’s input limit, or English chunks that waste it. Size chunks in tokens.
• Output limits bite first. A max_tokens of 512 is a comfortable English paragraph and a truncated Bengali one. Truncation shows up as a finish_reason of length, not as an error.
• Never split inside a cluster. A chunk boundary that lands between a consonant and its following hasanta produces two fragments that render as broken glyphs and embed as noise. Segment on grapheme clusters, not on code points.