Testing a Guardrail's False-Positive Rate on Legitimate Requests

Every figure on this page is derived from inputs named in the sentence that uses it. Substitute your own counts and prices; the arithmetic is the point, and the arithmetic has a wider error bar than most teams reporting this number realise.

What the corpus has to be

A false-positive rate is the proportion of legitimate requests a guardrail blocks. Everything rests on the word legitimate, so the corpus has to be labelled by a human, and the labelling has to happen without seeing the guardrail’s verdict. A corpus assembled by taking blocked requests and asking “was this really bad?” measures the reviewer’s agreement with the guardrail, which is a different quantity and always looks better.

• Sampled from production, not written. Hand-written “normal” requests are systematically politer, shorter and better spelled than real ones, and the awkward real ones are exactly what trips a filter.
• Randomly sampled within a period. A convenience sample from one team’s traffic gives that team’s rate.
• Redacted before it is stored. This corpus is a durable file of customer text. Redact at collection time, and see testing that a redacted fixture still reproduces the bug for the check that redaction did not destroy the signal.
• Held out. If the same corpus is used to tune the guardrail, its measured rate falls toward zero without the real rate moving at all. Keep a split that tuning never sees.

The rate, derived

Assume a corpus of 500 requests, each labelled legitimate by a reviewer who did not see the verdict. Assume the guardrail blocks 7 of them. Both numbers are assumptions for this worked example, not measurements.

false positives      = 7
legitimate requests  = 500
false-positive rate  = 7 / 500 = 0.014 = 1.4%

The cost of producing that figure is worth deriving too, because it decides how often you can afford to rerun it. Assume each request is sent to a guardrail model with an average of 600 input tokens including the guardrail’s own instructions, and assume a price of $0.25 per million input tokens — an assumed figure, not a quote; substitute your provider’s current published price.

tokens per run  = 500 requests x 600 tokens = 300,000 tokens
                = 0.30 million tokens
cost per run    = 0.30 x $0.25 = $0.075

Daily on every commit, 30 commits/day:
                  30 x $0.075 = $2.25/day = about $68/month

At that scale the constraint is not money, it is the labelling. Five hundred human labels is a day of somebody’s work, and the next section is about why that day may buy you less than you think.

The interval nobody reports

1.4% is a point estimate from 7 events. The uncertainty on it is large, and reporting it without an interval leads teams to react to movement that is noise. Use the Wilson score interval, which behaves properly for small counts where the textbook normal approximation does not. With p = 0.014, n = 500 and z = 1.96 for 95%:

z^2 = 3.8416

centre = (p + z^2 / 2n) / (1 + z^2 / n)
       = (0.014 + 0.0038416) / (1 + 0.0076832)
       = 0.0178416 / 1.0076832
       = 0.01771

margin = (z / (1 + z^2/n)) x sqrt( p(1-p)/n + z^2/(4n^2) )
       = (1.96 / 1.0076832) x sqrt( 0.0000276 + 0.0000038 )
       = 1.945 x 0.005604
       = 0.01090

95% interval = 0.0068 to 0.0286  ->  0.7% to 2.9%

So the honest statement is “somewhere between about 0.7% and 2.9%”. A release that moves the measured figure from 1.4% to 2.2% has not demonstrated a regression: the intervals overlap heavily. This is the single most useful thing on this page, because tracking a bare percentage over releases produces a chart that looks like signal and is not.

Two consequences follow. First, the rule of three: if zero out of 500 are blocked, the 95% upper bound is approximately 3 / n = 3 / 500 = 0.6%, so “no false positives” still means “could be as high as one in 167”. Second, to get a half-width of half a percentage point around a true rate near 1.5% you need roughly

n = z^2 p(1-p) / d^2
  = 3.8416 x 0.015 x 0.985 / (0.005)^2
  = 0.056760 / 0.000025
  = 2,270 labelled examples

Around 2,300 labels, not 500. That is the real price of tracking this number release over release, and it is worth knowing before committing to the metric.

Turning a rate into people

A percentage is not actionable. The number a product owner can act on is how many real users hit a wall today. Assume 40,000 legitimate requests per day, again an assumption:

blocked legitimate requests/day = 0.014 x 40,000 = 560
using the interval bounds        = 0.0068 x 40,000 = 272
                                   0.0286 x 40,000 = 1,144

Between roughly 270 and 1,150 legitimate requests blocked per day. That framing changes the conversation, because 1.4% sounds like a rounding error and 560 blocked customers does not. It also changes what you measure next: those 560 requests are a sampling frame, and reviewing fifty of them tells you which claim is over-firing far faster than moving the aggregate.

The same conversion is what makes a threshold arguable rather than arbitrary. A team that says “we accept up to 1%” is usually saying it without having converted 1% into 400 people a day, and the conversation changes once they do. It also fixes the direction of the test: the gate is on the daily count you can staff an appeals path for, and the percentage is the thing you derive from it, not the other way round.

Why the aggregate lies

A single rate averages over populations that behave nothing alike. The same guardrail can be near-zero on English-language billing questions and badly over-sensitive on a security team pasting exploit strings into a support ticket, on clinical vocabulary, or on a language the guardrail prompt was never written for. Averaged together those produce a comfortable aggregate and a specific segment that cannot use the product.

Stratify the corpus and report per stratum: language, product area, and whether the request contains user-pasted content. Each stratum needs its own sample size to have a usable interval, which is the practical reason to sample stratified rather than uniformly — 100 examples in each of five strata is far more informative than 500 drawn in proportion to traffic, where the smallest segment contributes eleven examples and an interval spanning most of the range.

Track the strata, not the average, and pair the result with the blocking-side suite in testing that a guardrail blocks what it claims to. A false-positive rate reported without the corresponding coverage figure can always be improved by blocking less, and usually is.