Testing for Refusal Rate Changes Before Committing to a Migration

Two models with comparable benchmark scores can decline different requests, and the requests they disagree about are almost never the obviously dangerous ones. They are the medical question phrased as a personal one, the security tooling request, the aggressive marketing copy, the legal summary. If your product lives near one of those edges, the refusal rate is a product metric, and you should have it for both models before you move traffic.

Why the rate moves at all

A refusal is not one thing. At least three separate systems can produce one, and they migrate independently:

• The model’s own post-training. Refusal behaviour is trained in alongside instruction following. Where the boundary sits, and how the model responds when it is unsure, comes out of that recipe. Two models from different lineages have different boundaries even when both vendors describe their policy in similar language. This is the part that follows the model family, not the vendor.
• A separate classifier in front of or behind the model. Some providers run input and output through a moderation stage that can block a request the model would have answered. This layer has its own thresholds and its own categories.
• Your own system prompt. A safety preamble tuned to loosen one model’s caution can tighten another’s, or be read as an instruction to be cautious in general. A prompt that ends with “decline anything you are unsure about” is a lever whose gain differs per model.

Because these are three systems, the direction of the change is not predictable from vendor reputation. A migration can raise the refusal rate on one category of your traffic and lower it on another in the same run. That is the case for measuring per category rather than in aggregate.

There is no refusal field you can rely on

The first thing people build is a check on a status field, and it catches the minority of cases. The signals that exist are real but partial:

• An OpenAI-shaped Chat Completions response can carry finish_reason: "content_filter", and the assistant message object has a refusal field that is populated when the model declines under structured outputs rather than emitting schema-shaped content. The full set of finish_reason values is documented separately.
• An Anthropic-shaped response reports its termination in stop_reason, and that enum has grown over time — treat any value your code does not recognise as a signal to inspect rather than as a normal completion.
• A blocked request may not return a completion at all. It may be an HTTP error from a moderation endpoint with its own body shape.

The dominant case is none of those. It is a perfectly ordinary successful completion, terminated normally, whose text is a polite decline. No field distinguishes it from an answer. So refusal detection is a text classification problem, and your harness needs a classifier.

Build it as a two-stage thing. A cheap lexical pre-filter catches the obvious shapes (an opening clause of apology, a modal decline, a referral to a professional) and a model-graded pass adjudicates the rest against a written rubric. Then hand-label a random sample of the classifier’s own output — a couple of hundred rows — so you know its error rate. A refusal classifier that is itself 8% wrong cannot detect a 3% shift, and that is the failure that makes the whole exercise decorative.

Building the set from real traffic

A set of prompts you invented tests your imagination. The set has to come out of your logs, because the whole question is what happens to your users’ requests.

1. Pull a window of production prompts long enough to cover a weekly cycle. Redact before anything else touches them.
2. Score every prompt for how close to the boundary it is. The cheapest usable score is the moderation category confidence from a classifier, but a keyword pass over your own domain’s sensitive vocabulary works and is transparent.
3. Stratify. Take the top slice by boundary score, but also take a uniform random slice — you need the false-positive side, where the new model refuses something plainly benign, and that never appears in a set built only from borderline prompts.
4. Bucket by the category that matters commercially: for a health product, symptom questions versus dosage questions; for a security product, defensive tooling versus offensive. Report per bucket.
5. Freeze it, version it, and store the full request as sent — including the system prompt and every parameter. A refusal test run against a different system prompt than production is measuring something else.

How big the set has to be

This is where most refusal comparisons go wrong: someone runs 50 prompts, sees three refusals become five, and reports a 67% increase. That number is noise.

For a proportion p measured over n independent prompts, the standard error is sqrt(p * (1 - p) / n), and a rough 95% interval is ±1.96 standard errors. Substitute your own expected base rate. At p = 0.05 and n = 50, the standard error is about 3.1 percentage points and the interval spans roughly 0 to 11% — wide enough to contain almost any result you might get. At n = 400 it falls to about 1.1 points, giving an interval of roughly ±2.1 points. Comparing two models doubles the variance of the difference, so the interval on the delta is wider still by a factor of about sqrt(2).

The practical reading of that arithmetic: a few hundred prompts per bucket lets you detect a shift of a few percentage points, and nothing smaller. If the shift you care about is one point, you are in thousands, and you should instead run the comparison on live shadow traffic where the volume already exists. Decide which regime you are in before you build the set, not after you get an ambiguous answer.

Running it and setting the gate

1. Run the frozen set against the current model first and record it as the baseline. You are looking for a delta, and without a same-harness baseline you are comparing against a memory.
2. Hold everything constant except the model string and whatever parameter renaming the target API forces. Note that pinning temperature to zero reduces variance but does not remove it — see why temperature zero is not deterministic. Run each prompt more than once if your budget allows and record the per-prompt disagreement rate; it tells you how much of the delta is sampling.
3. Classify every response into answered / hedged / refused, store the raw text alongside the label, and compute the rate per bucket with its interval.
4. Read the disagreements individually. The aggregate rate tells you whether to worry; the list of prompts that flipped tells you what to change. Frequently the fix is one clause in the system prompt rather than abandoning the migration.
5. Turn the result into a gate. Store the per-bucket baseline in your repo and fail the migration branch if any bucket moves by more than the interval you computed. That is what makes this a test rather than a one-off investigation, and it keeps working for the next model swap.

Keep the set after the migration. Refusal boundaries move within a model version too, so the same harness is what detects a silent model update that changes what your product will answer.