Why African Languages Are Underrepresented in AI Training Data

There are around two thousand languages in Africa and something over a billion people. Neither number is reflected anywhere in a training corpus. The interesting question is not whether there is a gap — the gap is easy to see — but which of several plausible causes is actually doing the work.

The explanation that is wrong

The explanation that circulates informally is that African languages are somehow harder: too many of them, too much variation, too little standardisation, tonal systems, unusual morphology. Every clause of that is either false or irrelevant.

Tone is not an obstacle to a text model; Mandarin is tonal and is among the best-supported languages in every system. Rich morphology is not an obstacle either; Finnish and Turkish are agglutinative and are handled adequately. Orthographic variation is real but not distinctively African — Norwegian has two written standards, Chinese has two character sets, and both are handled. And “too many languages” explains nothing about why the largest ones, with speaker populations in the tens of millions, are also missing.

If language-inherent difficulty were the cause, you would expect the quality gap to track linguistic properties. It does not. It tracks corpus size, which tracks something else entirely.

The counterexample that settles it

Take the comparison that makes the argument hard to escape. Icelandic has a few hundred thousand speakers. Estonian has about a million. Both are morphologically complex, both are spoken in one small country, and both are supported noticeably better by large models than Hausa, Yoruba or Amharic, each of which has tens of millions of speakers.

You can verify the input side of this yourself on the W3Techs content-language survey or Common Crawl’s per-crawl language statistics: languages with populations two orders of magnitude smaller sit higher in the web-text ranking. Whatever explains that, it is not the number of people who speak the language, and it is not the grammar.

Online, but not in your language

Here is the mechanism, and it is the part usually collapsed into the phrase “digital divide”. That phrase suggests African speakers are simply not online, which is decreasingly true — the ITU’s annual connectivity statistics show African internet use rising steeply, even as it lags other regions (ITU Facts and Figures). Hundreds of millions of people are online. The corpus did not grow proportionally.

The reason is that going online and producing text in your first language are different acts, separated by an education system. Across much of the continent, formal schooling, higher education, government administration, business and journalism are conducted in English, French, Portuguese or Arabic. A Nigerian graduate who speaks Yoruba at home was educated in English, types in English, and writes their blog post, their product review and their support ticket in English. Their contribution to the web is real and it is English-language.

So online presence does not convert into first-language corpus. The quantity that matters is not “speakers with internet access” but “text published in this language, in a crawlable form”, and the education system sits between the two, suppressing the conversion. That is a specific, historically contingent mechanism — the languages of instruction were set by colonial administrations and largely retained after independence — and it predicts exactly the pattern observed: the gap is worst for languages whose speakers are most integrated into the formal economy in a second language.

What the corpus pipeline discards

A second mechanism operates downstream, and it makes the first one worse. The text that is produced in African languages is often produced in places the corpus pipeline does not reach or does not keep:

• Closed platforms. Much first-language written communication happens in messaging apps and closed groups, which are not crawlable at all.
• Radio and oral media. Enormous volumes of first-language content are broadcast rather than written. A text corpus cannot see any of it without a transcription effort nobody has funded.
• Language-detection failure. Detectors are trained on the languages that already have data. Text in an unsupported language gets labelled as its nearest supported neighbour or discarded, so absence reproduces itself.
• Quality filters. Web-scale datasets discard pages failing heuristics tuned on English. Code-switched text — which is the normal register in much of Africa — scores badly on perplexity-style filters and is thrown out precisely because it is multilingual.

That last one deserves emphasis, because it is self-inflicted. The filtering step that improves English data quality actively removes the most characteristic African web text. Handling it is a separate design problem — see handling code-switched prompts.

The evidence that it is not the languages

The strongest evidence for the supply-side account is what happens when the supply changes. The Masakhane community’s participatory research produced translation and named-entity datasets for dozens of African languages through distributed volunteer effort (Nekoto et al., Findings of EMNLP 2020, Adelani et al., TACL 2021), and models trained and evaluated on that data perform in line with what their data volume predicts. There is no residual difficulty term. Add data, get quality; the languages behave like any others.

That reframes what the shortfall is. It is not a research problem waiting on an architectural breakthrough. It is a collection, licensing, funding and infrastructure problem, and the actions that address it are correspondingly mundane: digitising broadcast archives, publishing openly licensed text, funding annotation work at rates that make it a job, and building evaluation sets so that improvement is visible. The practical version for anyone building rather than researching is in crowdsourcing training data for a low-resource language.