Why Content Moderation Misses More in Low-Resource Languages

The gap in moderation quality between English and most other languages is not a decision anyone made. It falls out of how the training labels are produced, and it will not close by choosing a better model — because the constraint is not in the model.

The claim

A content moderation classifier is a supervised model. Its ceiling is set by labelled examples of the thing it is meant to catch, in the language it is meant to catch it in. Producing one such example requires a person fluent in that language and that variety, working from a policy document translated and calibrated for that context, at a cost per label that is roughly constant across languages. The number of labels a platform buys for a language therefore tracks that language’s commercial value to the platform, not its number of speakers and not the harm occurring in it.

That is the whole argument. Everything below is evidence that the mechanism is real and that the usual workarounds do not route around it.

Where moderation labels come from

The public datasets are the visible part, and their language coverage is a reasonable proxy for the private ones, because they were built by the same kind of process.

• RealToxicityPrompts (Gehman and colleagues, 2020) — around 100,000 prompts, English only, and for years the default toxicity benchmark.
• RTP-LX (Microsoft, 2024) — a multilingual counterpart in 28 languages, notable because the items were transcreated by humans rather than machine-translated.
• PolygloToxicityPrompts (2024) — a large prompt set covering 17 languages.

Set the best of those against the denominator. Ethnologue counts roughly 7,100 living languages; several hundred have more than a million speakers. Twenty-eight languages is a substantial engineering achievement and it is well under one per cent of the world’s languages, concentrated in exactly the ones that already have large digital corpora. The pretraining side shows the same shape: Common Crawl, the base of most open web corpora, publishes language statistics in which English is close to half of all detected content.

Note what that means for the harm side specifically. Even a language with a large pretraining share can have almost no labelled moderation data, because labels are commissioned and crawls are collected. The two curves are not the same curve, and the labelled one is much steeper.

Why cross-lingual transfer fails for this task

The standard rebuttal is that multilingual encoders transfer. They do, for some tasks. Topic classification transfers well: whether a message is about football is a semantic property, and a shared multilingual representation captures semantics across languages reasonably.

Moderation is not that kind of task. Whether a term is a slur is a lexical and social fact about a specific community, not a semantic one. A model can know perfectly well that a word denotes an ethnic group and have no signal at all that this particular word, in this particular country, is the pejorative rather than the neutral term. There is nothing in the semantics to transfer, because the offensiveness is not in the semantics.

Three further properties resist transfer for the same reason. Reclaimed terms invert depending on who is speaking. Coded language and euphemism are invented locally and change on a timescale of weeks. And the same word can be a slur in one national variety and unremarkable in another, which means a single classifier for “Spanish” or “Arabic” is already an aggregation error before any language gap is considered.

Then there is text that belongs to no single language column at all. Romanised and code-switched writing — Hinglish, Arabizi, Spanglish — falls outside both the English model and the native-script model, because it looks like Latin script to the language detector and does not resemble English to the classifier. See handling Hinglish text and Arabizi and numeral substitution for what that does in practice.

The feedback loop that keeps it that way

The gap is self-sustaining, which is why it does not slowly close. Moderation training data is largely produced from enforcement: reports, reviewer decisions, appeals. A weak classifier in a language surfaces fewer items for review, which produces fewer decisions, which produces fewer labels, which keeps the classifier weak. Users who see that reports go nowhere report less, removing the only external signal that would break the loop.

Measurement closes the circle. Precision and recall are reported where there is a labelled evaluation set, and the languages without training labels are exactly the languages without evaluation labels. The gap is therefore not visible in the dashboard that would justify closing it. The same structural blindness runs through public evaluation generally — benchmark coverage gaps by language is the general case.

The obvious objection

“Large language models are multilingual. Prompt one as the moderator and the problem goes away.” This is now the standard proposal and it is weaker than it looks, for three reasons.

First, it inherits the same skew. The model’s own judgement of what is harmful in Swahili comes from its alignment data, which is subject to the identical labelling economics. Using an unaudited system as the auditor does not create information.

Second, calibration does not survive the language change. A threshold tuned to a target precision on English does not deliver that precision in another language, because the score distribution is different. A single global threshold across languages guarantees that every language except the one it was tuned on is operating at an unintended point — and you cannot tell which direction without per-language labelled data, which is the thing you did not have.

Third, cost. Platform-scale moderation is billions of items; a model call per item is a different order of expense from a classifier inference, so in practice the model is used as a second stage behind a cheap first-stage filter. If the first stage has no coverage in a language, the second stage never sees the item, and the model’s multilingual ability is irrelevant.

What to build instead

The design conclusion is not “try harder”. It is to stop representing absence of coverage as absence of harm.

• Make “no coverage” a distinct outcome. A pipeline should return covered-and-clean, covered-and-flagged, or not-covered — three states. Collapsing the third into the first is the single decision that turns a known limitation into an invisible one.
• Set thresholds per language. Even a small labelled set per language, a few hundred items, is enough to place an operating point deliberately rather than inherit one.
• Report per language, never aggregated. An aggregate recall figure is dominated by the traffic mix, so it improves when English traffic grows.
• Route by coverage, not by confidence. Send uncovered languages to human review or to a stricter default, because a low score from a model with no competence in a language is not evidence of anything.
• Buy labels for the languages you actually serve. A few thousand well-annotated items in a language is a small budget line and the only intervention that moves the constraint.