Why Embedding Quality Is Worse for African Languages

“Embeddings are worse for African languages” is true and almost useless. This page gives the sourced version: what has been published, what has not been measured at all, and how much of the gap the arithmetic of corpus share and tokenizer fertility accounts for.

What the published record says

The standard reference for how unevenly language resources are distributed is Joshi, Santy, Budhiraja, Bali and Choudhury, “The State and Fate of Linguistic Diversity and Inclusion in the NLP World”, published at ACL in 2020. It sorts the world’s languages into six classes, 0 to 5, by how much labelled and unlabelled data exists for each. Class 5 — the languages with abundant resources of both kinds — holds seven members. The overwhelming majority of the roughly seven thousand living languages sit in class 0, with essentially nothing. Almost every African language other than a handful of the largest is in the bottom two classes in that paper’s table.

On the benchmark side, the gap is documented by the fact that the benchmarks had to be built recently and by hand. MasakhaNER, published by Adelani and a large collaboration in 2021 — “MasakhaNER: Named Entity Recognition for African Languages” — covered ten African languages, and its 2022 successor extended the set to twenty. Meta’s No Language Left Behind project, described in “No Language Left Behind: Scaling Human-Centered Machine Translation” (2022), built the FLORES-200 evaluation set specifically because evaluation data did not exist for most of the two hundred languages it targeted. When a benchmark has to be created before a question can be asked, the answer to “how good were models before?” is that nobody knew.

For embeddings specifically, the reference leaderboard is MTEB, introduced by Muennighoff, Tazi, Magne and Reimers in “MTEB: Massive Text Embedding Benchmark” (2022). Its multilingual successor, MMTEB, was published in 2025 and widened language coverage substantially. Check the current task list on the MTEB repository rather than a figure from an article: whether your language has a retrieval task, and how many documents that task contains, is the thing that decides whether a leaderboard position means anything for you.

The corpus share, and why nobody should quote it

The usual next move is to quote a per-language byte count from a web crawl. This page will not, and the reason is worth stating plainly: those figures circulate widely, disagree with each other, are usually reported without saying which crawl or which filtered derivative they came from, and change with every crawl. A number that cannot be pinned to a snapshot is not a fact.

What can be said, and checked: Common Crawl publishes a language breakdown for each crawl, derived from automatic language identification, at its own statistics site. English is by a wide margin the single largest language there, and African languages sit far down the long tail. Read the current plot before you cite anything. Note also that the identification is automatic, so the tail is doubly unreliable: a language detector trained mostly on high-resource languages misclassifies low-resource ones, which understates them further.

A second, structural point that does not move: web crawl share is a poor proxy for speaker population. Bengali, Swahili, Hausa, Yoruba and Amharic each have tens of millions of speakers, and the volume of text on the web in those languages is not proportional to that. The gap is about publishing infrastructure, keyboard availability, the dominance of a colonial language in formal writing, and the fact that a great deal of the digital text that does exist lives in messaging apps rather than on crawlable pages.

Turning a corpus share into a token count

Rather than quote a figure, here is the derivation, with every assumption labelled. Substitute your own numbers.

ASSUMPTIONS (replace with values you can verify)
  T   = total training tokens for the model        = 2e12   (assumed)
  s_e = share of the mix that is English           = 0.45   (assumed)
  s_x = share of the mix that is language X        = 0.0005 (assumed)

DERIVED
  English tokens seen   = T * s_e = 9.0e11
  Language X tokens seen = T * s_x = 1.0e9
  Ratio                  = s_e / s_x = 900x

Nine hundred to one is the headline, and it is already an understatement, because the comparison that matters is not tokens but distinct contexts per word. A representation is learned from the variety of contexts a unit appears in. If language X gets a thousandth of the data, its vocabulary items appear in a thousandth of the contexts, and the rarer half of its vocabulary appears in almost none. Quality does not degrade uniformly across a language; it degrades from the tail of the vocabulary inwards, which is why a model can look competent on simple sentences in a low-resource language and fall apart on domain-specific text.

The tokenizer takes a second cut

The merge table is learned from the same skewed mix, so it allocates its vocabulary slots to what it saw. A language with little representation gets few dedicated units and is encoded as short byte fragments. The measurable form of this is fertility: tokens per word.

ASSUMPTIONS (measure these yourself with your model's tokenizer)
  f_e = tokens per word, English      = 1.3  (assumed)
  f_x = tokens per word, language X   = 4.0  (assumed)

DERIVED
  Words seen, English   = 9.0e11 / 1.3 = 6.9e11
  Words seen, language X = 1.0e9 / 4.0 = 2.5e8
  Ratio in words        = 2770x   (vs 900x in tokens)

Fertility multiplies the gap rather than adding to it. It also costs you twice more at inference: the same passage consumes several times more of the context window and of your budget, and a chunk sized in tokens holds several times less meaning. That second effect is the one that quietly ruins retrieval quality, because your chunks contain less content than you think they do.

Fertility is the single cheapest diagnostic on this page and it takes one line to compute. Tokenize a hundred words of your language, divide. Many African languages compound the problem with diacritics that carry meaning — Yoruba marks tone with combining accents, and Ewe, Igbo and many others use letters outside the basic Latin set — so the same word typed with and without its marks tokenizes differently and, without normalization, indexes as two different words.

What to check for your language

1. Measure tokenizer fertility for each candidate model on a hundred words of real text from your domain. A fertility above three or four is a warning about cost and chunking regardless of retrieval quality.
2. Check whether the language has any retrieval task in MTEB at all, and how large it is. A leaderboard row backed by a few hundred documents is a weak signal.
3. Normalize to NFC and decide explicitly what happens to tone marks and extended Latin letters, on documents and on queries alike.
4. Build a small labelled set of your own. It is the only way to get a number for a language with no benchmark, and it is a day of work: how to build one from scratch.
5. Keep a lexical retrieval path. For a language the model was barely trained on, exact term matching is not a fallback, it is a large part of your recall.