Building a Content Safety Layer That Isn't Useless

No precision or recall numbers appear on this page, and that is the argument rather than an omission. A safety threshold depends on your content distribution and your base rate, so a number from someone else’s deployment is not evidence about yours. What transfers is the method.

How safety layers actually fail

They rarely fail by missing something dramatic. They fail in one of three quieter ways:

• Tuned so tight that users route around it. The filter blocks legitimate work, the team adds an exception, then another, and eventually a bypass flag that becomes the default. The control is nominally live and effectively off.
• Tuned so loose it is decoration. Nobody measured, so the threshold is whatever the example in the documentation used. It catches the obvious and provides an assurance nobody has tested.
• Unmeasured, and therefore unchangeable. With no labelled set, every proposed adjustment is an argument between anecdotes, so nothing is adjusted and the layer ossifies.

The common cause is the absence of a labelled evaluation set. Almost everything else follows from fixing that.

Building that set is where the real difficulty sits, and it is worth expecting: your labellers will disagree, sometimes on a third of the hard cases. That disagreement is data rather than noise. If two careful people cannot agree whether an item violates the policy, no classifier threshold will resolve it, and the correct fix is upstream — write the policy more precisely, or route that category to a middle action rather than to block-or-allow. Measure agreement explicitly before you measure the classifier, because a model can never score better than the consistency of the labels you graded it against.

The four numbers, and the one that misleads

Every classification decision is one of four outcomes: true positive (harmful, blocked), false positive (benign, blocked), true negative (benign, allowed), false negative (harmful, allowed). From those:

• Precision = TP / (TP + FP). Of what you blocked, how much deserved it. Low precision is what makes users hate the product and campaign for the bypass flag.
• Recall = TP / (TP + FN). Of what deserved blocking, how much you caught. Low recall is what the incident review is about.
• Accuracy = the proportion of all decisions that were correct — and it is nearly useless here. If 0.1% of your traffic is harmful, blocking nothing at all scores 99.9%. Never report accuracy for a safety layer; it is the number that makes a broken filter look excellent.

Precision and recall trade against each other as the threshold moves, and there is no correct point in the abstract — only a point that reflects your relative cost of the two errors. Write that ratio down explicitly. A public-facing surface with regulatory exposure and an internal drafting tool should land in obviously different places, and if your two surfaces share a threshold, at least one of them is wrong.

Why a borrowed threshold is worthless

Suppose a vendor reports a classifier with 95% recall and 90% precision. On traffic where 1 in 1,000 items is harmful, the precision you experience is not 90%: of 100,000 requests, 100 are harmful, you catch 95, and the false positives come from the 99,900 benign ones. At even a 1% false-positive rate that is 999 wrongly blocked users against 95 correct blocks — a precision under 10%, from a classifier whose published numbers were good.

Nothing was dishonest in the vendor’s figure. It was measured on a balanced set, and your base rate is not balanced. This is the base-rate fallacy, and it is the reason the only threshold worth shipping is one measured on traffic shaped like yours.

A threshold sweep you run on your own data

Assemble a labelled set first: a few hundred items sampled from your own real traffic, labelled by two people with disagreements resolved by a third. Include the hard cases deliberately — security discussions, medical questions, fiction, quoted abuse, other languages — because the easy cases tell you nothing you did not know. Then sweep:

type Labelled = { text: string; harmful: boolean };

type Row = {
  threshold: number;
  tp: number; fp: number; tn: number; fn: number;
  precision: number; recall: number; blockRate: number;
};

export async function sweep(
  set: Labelled[],
  score: (t: string) => Promise<number>,   // your classifier, 0..1
  thresholds = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
): Promise<Row[]> {
  // Score once. Sweeping the threshold is free; re-scoring is not.
  const scored = await Promise.all(
    set.map(async (s) => ({ ...s, score: await score(s.text) })),
  );

  return thresholds.map((threshold) => {
    let tp = 0, fp = 0, tn = 0, fn = 0;
    for (const s of scored) {
      const blocked = s.score >= threshold;
      if (blocked && s.harmful) tp++;
      else if (blocked && !s.harmful) fp++;
      else if (!blocked && !s.harmful) tn++;
      else fn++;
    }
    return {
      threshold,
      tp, fp, tn, fn,
      precision: tp + fp === 0 ? 1 : tp / (tp + fp),
      recall: tp + fn === 0 ? 1 : tp / (tp + fn),
      // The number that predicts your support load. Compare it against
      // the base rate: if you block 4% of traffic and 0.1% is harmful,
      // 39 in every 40 blocks are wrong.
      blockRate: (tp + fp) / scored.length,
    };
  });
}

Read the table by fixing the error you cannot tolerate and taking the best available value of the other. “Recall must be at least 0.95 for this category; what is the highest precision available at that recall?” is a decision. “Which row has the best F1?” is an abdication, because F1 weights the two errors equally and you almost never do.

Then re-run the sweep on every model change, prompt change and vendor change. That regression signal is worth more than the absolute numbers, and it is the thing that stops a routine model upgrade from silently changing your safety posture.

Operating it

• Use more than two outcomes. Allow, flag-and-log, soften, require-confirmation, block. Most items that score in the middle deserve one of the middle actions, and a binary filter forces every ambiguous case into the error you can least afford.
• Different thresholds per category and per surface. The threshold for self-harm content and the threshold for profanity have no reason to be the same number.
• Log every decision with its score. Without the score you cannot re-tune retrospectively, and re-scoring historical traffic is expensive.
• Give users a route. An appeal or feedback path is both the humane choice and your best source of labelled false positives, which is the data the sweep most needs.
• Sample what you allowed. False negatives never complain. A weekly human review of a random sample of allowed traffic is the only way they enter your numbers at all.
• Fail closed on the highest tier only. If the classifier is unavailable, blocking everything is an outage and allowing everything is a gap. Decide per surface, in advance, and write it down.