Why Bengali Support Lags Despite Its Enormous Speaker Population

Bengali has more speakers than German, French and Italian combined, and a model will handle it worse than any of them. That is not a paradox and it is not neglect by a single vendor. It is what happens when you use a population statistic to predict a corpus statistic.

The mismatch

Ethnologue places Bengali among the half-dozen most spoken languages in the world, with well over two hundred million first-language speakers across Bangladesh and the Indian states of West Bengal, Tripura and Assam, and a total above that once second-language speakers are counted (Ethnologue). It is the national language of a country of some 170 million people. It has a continuous literary tradition going back centuries and a Nobel laureate in literature.

Now look at the supply side. On the W3Techs survey of content languages across the top ten million websites (W3Techs) and on Common Crawl’s per-crawl language statistics (Common Crawl), Bengali sits below languages with a twentieth of its speakers. Check it against Dutch, Czech, Swedish or Greek on either source. The ordering is not close, and it is the ordering that predicts model quality.

Two numbers that measure different things

The core of the argument is a category error that is easy to make. Speaker population is a demand-side number: it tells you how many people would benefit from support. Corpus size is a supply-side number: it tells you how much text about anything, in that language, was published in a form a crawler could fetch, parse and identify.

Those two are connected only through a long chain, and the chain has several places where it can break: literacy in the language rather than in a second language of education; internet access; whether the platforms people use are crawlable; whether the writing is in a standard encoding; whether the pages carry a language tag a detector trusts; and whether the material is under a licence that survives the dataset-filtering step. A language can have a vast population and a weak link anywhere in that chain and end up with a corpus the size of a language spoken by two million people.

Bengali has a weak link, and it is unusually specific.

The decades of text nobody can read

Bengali publishing digitised early and digitised badly. For many years the dominant way to typeset Bengali on a computer was a family of custom fonts that mapped Bengali glyphs onto ASCII codepoints. The bytes on disk say “Latin letters and punctuation”; the glyphs on screen are Bengali; the mapping lives in the font file and nowhere else.

Consider what that does to every downstream step. A crawler fetches the page and gets a byte string of Latin characters. A language detector reads it and reports English, or garbage, or nothing. A corpus filtering pipeline — which for every large web dataset includes a quality heuristic that discards text failing to look like natural language — throws it away as noise. A deduplicator cannot match it against the same article in Unicode. A tokenizer, if it survived that far, would learn merges that correspond to no linguistic unit at all.

So a large body of Bengali journalism, government material and literature from the pre-Unicode era is, from the point of view of every corpus-building pipeline, not Bengali text. It is not missing from the web; it is invisible on it. This is a mechanism, it is checkable — open an older Bengali page and copy text out of it, and see what you get in the clipboard — and it does not appear in any account of Bengali support that stops at “low-resource language”.

Modern Bengali publishing uses Unicode, so this is a legacy problem that shrinks over time. But training corpora are built from historical web crawls, and the archive is exactly where the legacy lives.

What the script costs on top

A second, smaller effect compounds it. The Bengali script is an abugida with an extensive set of conjunct consonants, and a single visual cluster can be several codepoints: consonant, virama, consonant, vowel sign. Tokenizers trained predominantly on Latin text have few merges for these sequences, so Bengali fragments into many more tokens per word than English does. Petrov et al. documented tokenisation length disparities of more than an order of magnitude across languages in work presented at NeurIPS 2023 (Language Model Tokenizers Introduce Unfairness Between Languages).

That is a cost multiplier on every request — the token cost of Bengali works it through — but it is also a training effect. More tokens per unit of meaning means each unit of Bengali meaning occupies more of a fixed context window during training, so the model sees fewer complete Bengali documents for a given compute budget. And the same codepoint sequence can be normalised differently across sources, so identical words fail to match. Bengali needs NFC normalisation applied consistently before anything else touches the text.

What would actually close the gap

The useful conclusion is that the levers are not the ones people reach for. Asking a vendor to “support Bengali better” is asking them to solve a data-availability problem they do not control. The things that move the number are unglamorous: converting legacy-encoded archives to Unicode so that thirty years of published Bengali becomes machine-readable; producing openly licensed Bengali text in domains where it does not exist; and building evaluation sets, because capability follows measurement.

The same argument applies with different specifics to every large language that behaves like a small one, which is most of South Asia and much of Africa — why African languages are underrepresented runs the supply-side argument on a different weak link in the same chain. The general lesson is the one to keep: when someone predicts model quality for a language from its speaker count, they are using the wrong variable, and the error is not small.