Content Moderation by AI: Scale, Errors and Appeals

Every platform of any size makes a very large number of moderation errors in both directions, and this is not a sign that the classifiers are bad. It follows from prevalence and volume, and the interesting decisions are about which errors to prefer and what happens to the people on the wrong end of them.

What automated moderation actually is

Not one classifier. A pipeline, and the consequential choices are between the stages rather than inside them:

• Matching. Perceptual hashing against databases of known violating content. High precision, no judgement required, and the least controversial part — though the contents of the databases and who may add to them is itself a governance question.
• Classification. Models scoring content against policies. Produces a continuous score that somebody has to turn into an action with a threshold.
• Routing. Which items go to human review. Review capacity is finite, so this is a rationing decision, and it decides more outcomes than the classifier does.
• Action. Remove, restrict reach, age-gate, label, demonetise, suspend the account. Graduated actions allow lower confidence thresholds, which is the main way platforms manage the error trade-off — and reach reduction is also the least visible and least appealable action.
• Appeal. Where a wrong decision can be corrected, if the route exists and is reachable.

The arithmetic of scale

For a policy area with volume V and prevalence p, at an operating point with rates TPR and FPR:

  caught violations   =  V · p · TPR
  missed violations   =  V · p · (1 - TPR)
  wrongly actioned    =  V · (1 - p) · FPR

  share of actions that are wrong
        =        (1 - p) · FPR
          ------------------------------
          (1 - p) · FPR  +  p · TPR

For most policy areas p is very small — violating content is a
tiny fraction of everything posted — so (1 - p) ≈ 1 and the
wrongly-actioned term is driven almost entirely by FPR · V.

Two conclusions follow, and neither depends on the quality of any particular model. Because p is small, false positives can outnumber true positives even at a very low false positive rate, purely because they are drawn from a vastly larger pool. And because the threshold moves TPR and FPR together, there is no setting that reduces both error types — you are choosing a point on a curve, not finding a correct answer.

That choice is normative. Preferring to catch more violations at the cost of removing more legitimate speech, or the reverse, is a judgement about which harm is worse and who bears it — and the two harms fall on different people. Presenting it as a technical tuning parameter, which is how it is usually presented, hides a decision that ought to be explained and defended. The parties who would defend opposite settings are easy to name and both have a case: safety advocates point to real harm to victims from under-enforcement, and speech advocates point to the systematic silencing of legitimate expression from over-enforcement.

Context, not language, is the hard part

The persistent failures are contextual rather than linguistic. Counter-speech quotes the abuse it condemns. Documentation of atrocity contains the imagery a policy prohibits. Reclaimed slurs are used within a community. Medical, legal and educational content discusses acts that are prohibited to promote. Satire and irony invert surface meaning. Coded language shifts weekly, precisely because it is designed to.

These are not edge cases; they are a substantial share of the disputed decisions, and they are the cases where the correct answer requires knowing who is speaking, to whom, and why. That information is usually not in the item being classified.

The uneven part is resourcing rather than modelling. Classifier quality varies across languages and dialects with the data available, and human review capacity varies far more — the availability of reviewers with the right language and cultural context is the binding constraint in most markets, and it correlates with the commercial importance of the market rather than with the risk in it. This asymmetry has been documented repeatedly by researchers and civil society organisations and is arguably the single largest quality gap in the field.

What due process would require

The published articulation most people converge on is the Santa Clara Principles, developed by civil society organisations, and its core is modest: numbers, notice and appeal.

• Notice with a specific reason. Which rule, and what in the content triggered it. “Violated community guidelines” cannot be contested because it does not say what to contest.
• Disclosure that automation was used, and whether a human reviewed the decision.
• Appeal to a different decision-maker, within a stated time, with the ability to submit context.
• Reinstatement that restores reach and standing, not just the item — a restored post that nobody sees is a partial remedy.
• Reporting by policy area and language: actions taken, appeals lodged, appeals upheld. The overturn rate is the only externally visible signal of accuracy that exists.

The cost objection is real and should be stated rather than waved away. Appeals are human-expensive at a volume where automation was adopted because humans were unaffordable; adversaries can exhaust an appeals system deliberately; and in some categories delay is itself the harm. The strongest response is not that cost does not matter but that appeals capacity is a function of how aggressively the threshold is set, which brings the two halves of this page together: a platform that cannot afford appeals is telling you something about its threshold.

What language models change

Three things, of which the first is the least discussed and the most significant.

Policy becomes a prompt. Where a classifier had to be retrained on labelled data to change behaviour, a policy expressed in natural language can be edited and deployed the same day. That is a large operational gain and a governance risk of the same size: the effective speech rules of a large platform become editable without the labelling process, the review, or the record that a retraining cycle used to impose. Anyone assessing a moderation system should now ask who can change the prompt and what is logged when they do.

Explanations become generatable. A model can produce a fluent reason string for every decision, which addresses the notice problem in form. It must not be presented as the cause of the decision: as covered on the accountability page in this cluster, generated reasoning is not a faithful account of the computation, and a confident-sounding reason that is not the real one is worse for contestation than a terse accurate one.

Triage improves. Better ranking of what humans should look at first is a genuine gain, and it is the application with the clearest benefit and the least controversy, because it changes the order of review rather than removing it.