What Changes in Refusal Behaviour After a Model Migration

Support tickets say the assistant “keeps saying it can’t help” on requests it handled last week. Nothing in your code changed except the model. The first problem is that most refusals do not arrive with a flag on them, so before you can measure the change you have to decide what counts as one.

What the spike looks like

A refusal shift after a migration is rarely uniform. It concentrates in a few request shapes — a template that quotes user-supplied text, a category of document, a system prompt that assigns a role, a particular language. That concentration is the useful signal, because a shift that affected everything equally would be a prompt problem and a shift confined to one template is a content boundary you can see the edge of.

It also runs in both directions, and the direction nobody notices is the looser one: a model that declines less often will produce output on requests your product intended to decline, and no ticket is ever filed about that. If you are auditing a migration, check both tails.

Detecting a refusal at all

There are four detection surfaces and their availability differs by provider, which is itself the migration problem.

• A dedicated stop reason. Anthropic’s messages can carry a refusal value in stop_reason, distinct from end_turn. Where it is present it is unambiguous and free.
• A filter-driven finish reason. OpenAI’s chat completions define content_filter among the finish_reason values, and Google’s responses carry a safety-related finish reason plus prompt-level block feedback. These indicate a system-level block rather than the model declining in prose — a different event, and one you should count separately. The full value list is in OpenAI finish_reason values.
• A structured refusal field. Under structured outputs, a refusal can arrive as a separate field on the message rather than as content, so a client reading only content sees an empty string. If your migration produced empty completions rather than apologies, look for this before concluding the model returned nothing — handling empty completions is the adjacent test.
• Nothing at all. The commonest case. The model declines in ordinary prose, the stop reason is the normal one, and there is no field to read.

For that last case you need a classifier, and the important design decision is not to build it from a keyword list. Phrases like “I can’t help with that” are model-specific and language-specific, so a regex tuned on the old model undercounts the new one by construction — which manufactures exactly the delta you are trying to measure. Use a small model or a fixed judge prompt to label an output as refusal, partial refusal or compliance, pin it, and validate it against a few hundred hand-labelled examples drawn from both models before you trust a single number it produces.

Four kinds of refusal

• Flat refusal. No attempt, usually with a short explanation. Easiest to detect and the one users report.
• Partial compliance. The answer arrives with the sensitive part removed, or hedged into uselessness. Invisible to any detector and invisible in any aggregate; it shows up as a quality complaint months later.
• Over-caution. Disclaimers, requests for clarification, a suggestion to consult a professional. Not a refusal by any classifier, and a product regression all the same.
• System-level block. The request or the response was stopped by a filter outside the model, signalled by a finish reason. The remedy is configuration, not prompting, and it belongs with migrating a guardrails configuration.

Separating these matters because their fixes are unrelated: a flat refusal is often solved by context that establishes legitimacy, partial compliance by an explicit output contract, over-caution by an instruction about tone, and a block by a settings change or a different route.

Measuring the delta honestly

The comparison is only meaningful against a fixed prompt set with everything else pinned, and it should have been captured before the migration — testing the refusal rate before a migration is the page on doing it in advance. If you did not, you can rebuild a set from logged requests, provided you redact them and keep the sample representative rather than drawn from the complaints.

1. Assemble the set from real traffic, stratified by template and by request category, with enough per stratum that a handful of cases does not move the number. Include a control stratum you expect no refusals on, as a check on the classifier.
2. Pin everything except the model. Same system prompt after translation, same temperature, same token budget, same tools. A dropped system prompt is a common cause of a refusal spike all by itself, because the context that made the request legitimate was in it.
3. Run both models on the same set, several samples per prompt if temperature is non-zero, and label every output with the pinned classifier.
4. Report per stratum, with counts. A single percentage across the whole set is the least informative form of this result. The deliverable is which templates moved, by how much, on how many cases.
5. Read the disagreements. Cases refused by one and answered by the other, both directions. Twenty of these tell you more about the boundary than the aggregate does, and they are what you take to a decision.

Two cautions on interpretation. Do not report this as one model being safer or more capable: you measured a boundary against your prompt set, which is not a general property, and the direction that suits your product is not a virtue. And do not compare a number you compute now against one someone published elsewhere — different prompt sets, different classifiers, no comparability.

Reducing the rate

• Restore the context that establishes legitimacy. Role, purpose and audience in the system prompt do most of the work on a legitimate request that reads as borderline out of context. This is not jailbreaking; it is supplying facts the model needs.
• Separate data from instruction. Where user content is quoted into the prompt, mark it as content to be processed rather than as a request from the user. Requests that refuse because a quoted document sounded like an instruction are the largest fixable group, and the marking convention is itself model-specific — see rewriting XML-delimited prompts.
• State the output contract. A model that knows the expected form of a compliant answer is less likely to hedge its way out of producing one.
• Route the residue rather than fighting it. Where a stratum refuses persistently after prompt work, sending that stratum to a different model is a legitimate engineering answer, and it needs a fallback path that treats a refusal as a routable outcome rather than as success. That is the same machinery as falling back on a model error, with a different trigger.
• Re-run the fixed set after each change. Prompt edits aimed at refusals frequently move behaviour elsewhere, and the control stratum is what tells you.