Languages Missing Entirely From Public Training Corpora

“Most languages are underrepresented in AI training data” is true and useless. The number worth having is the count of languages with essentially no text in the public corpora at all — not a thin slice, nothing — because that is a different engineering situation with different options.

The denominator: how many languages there are

Every claim about coverage is a fraction, and the fraction is meaningless without its denominator. Ethnologue, the most widely cited catalogue, counted 7,164 living languages in its 26th edition, published in 2023. The open alternative, Glottolog, catalogues a comparable number of spoken and signed languoids, and the two disagree at the margins because “language” versus “dialect” is a judgement, not a measurement.

A second denominator matters just as much and is much smaller: the number of languages with an established written form. A language with no orthography in daily use cannot contribute text to a web crawl no matter how many people speak it, so the population of languages that could plausibly appear in a corpus is a subset of the 7,000, not the whole of it. Any page that divides a corpus’s language count by 7,164 and calls the result “coverage” is quietly comparing a written-text quantity against a spoken-language count.

Zero is a different category from low

The most useful published framing of this comes from Joshi and colleagues’ 2020 ACL paper The State and Fate of Linguistic Diversity and Inclusion in the NLP World, which sorts languages into six classes by how much labelled and unlabelled data exists for each. The extremes are the interesting part. Class 5, the languages with abundant data of both kinds, contained seven members: English, Spanish, German, Japanese, French, Arabic and Chinese. Class 0 — the paper calls them the Left-Behinds, meaning languages with essentially no usable unlabelled text and no labelled data at all — contained 2,191.

That 2,191 is the figure to carry, with its date attached. It is not “these languages are hard”. It is “there is nothing to train on”, which rules out the ordinary remedies. You cannot fine-tune your way out of class 0, you cannot few-shot your way out of it, and you cannot evaluate your way out of it either, because evaluation needs held-out text that also does not exist. The gap between class 0 and class 1 is qualitative; the gap between class 3 and class 4 is merely a matter of degree.

What the corpora themselves claim to cover

The public corpora publish their own language lists, which is the honest place to check. The broad web crawls that underpin most pretraining sets are filtered by language identification into a few hundred buckets at most: Common Crawl derived sets such as CC-100 and mC4 land at roughly one hundred languages apiece. Meta’s No Language Left Behind work, published in 2022, deliberately pushed further and released parallel data and the FLORES-200 evaluation set across a little over two hundred language varieties — the largest single expansion the field has had.

Two hundred against seven thousand is the shape of the problem, and it is worth stating what the two hundred is not. It is not two hundred languages with pretraining-scale corpora. It is two hundred languages for which somebody assembled enough parallel sentences to train and evaluate translation. The volume behind the two-hundredth entry and the volume behind English differ by many orders of magnitude, which is why a model can nominally “support” a language and still produce fluent nonsense in it.

Why the gap does not close on its own

Three mechanisms hold it open, and none of them is a funding problem that a larger crawl fixes.

• Language identification is itself a low-resource task. A crawl can only file text under a language its classifier knows. A language absent from the classifier’s label set is not filed as that language — it is filed as a neighbour, or discarded as noise. Absence from the pipeline’s language list therefore causes absence from the corpus, which then justifies the language staying off the list.
• Quality filters preferentially delete small languages. Perplexity filters, deduplication and heuristic cleaning are tuned on high-resource text. Text in a language the filter’s reference model has never seen scores as gibberish and is removed by exactly the step that is supposed to be improving the corpus.
• Speakers publish in a second language. Where a community is bilingual with a dominant language, the written output that reaches the web is often in the dominant one. Digital text volume tracks writing habits and institutional publishing, not speaker population — which is why speaker counts predict corpus size so poorly.

How to check a specific language

For one language, the question is answerable in an afternoon and you should answer it rather than trusting a summary. Take the ISO 639-3 code, then look for it in the published language lists of the corpora and datasets themselves: the FLORES-200 language list, the mC4 and CC-100 language tables, the Wikipedia list of Wikipedias by article count, and the OPUS parallel-corpus index. A language present in FLORES-200 with a Wikipedia of a few thousand real articles is class 1 or 2 territory. A language present in none of them, with no Wikipedia or a machine-generated one, is class 0 in practice regardless of what any model’s marketing page lists.

The practical consequence is that a class 0 language is a data-creation project before it is a modelling project, and the sequencing is not optional. That is the same conclusion the building for a language with no digital corpus page starts from, and it is why low-resource output degrades into confident invention rather than into visible failure — there is no held-out data for the model to be visibly wrong against.