Migrating a Prompt's Refusal-Handling Fallback Logic

Somewhere in your codebase is a line that looks like if "I'm sorry" in text.lower():, and it decides whether to fall back to a canned answer, escalate to a human, or retry on another model. After a migration it either stops firing or starts firing on everything, and both failures are silent.

The literal symptom

There are three shapes this arrives in. The first is a fallback that never triggers: users start seeing raw decline text where they used to see your handled message, because the new model declines with different wording. The second is the opposite — the fallback fires on perfectly good answers, because the new model happens to open more replies with an apology or a caveat, and every one of them matches your substring.

The third is a crash, and it is the most informative:

AttributeError: 'NoneType' object has no attribute 'lower'
  at classify_response(resp)  ->  resp.choices[0].message.content.lower()

That happens because on an OpenAI-shaped API the assistant message’s content can be null when the model declined through a dedicated field instead of through prose. The traceback is telling you exactly what the fix is: the refusal was reported structurally, and your code went looking for it in the text.

Why phrase matching was never an interface

Refusal phrasing is a sampling artefact. It changes with the model version, with the system prompt, with temperature where temperature is still accepted, and with the output language — a detector written against English apologies fails the day you serve a Spanish-speaking customer, migration or no migration. It also changes with prompt tuning: an instruction telling the model to avoid preambles will remove the very phrase the detector keys on, so a routine prompt improvement breaks an unrelated fallback path.

The migration does not create this fragility. It reveals it, all at once, on a day when several other things also changed — which is why it is worth fixing structurally rather than by updating the phrase list.

The structural signals, per API shape

Each API family reports a decline in a terminal field, and there are three places to look. All three should be handled, because a multi-provider deployment will encounter all three.

• Messages API. A successful HTTP 200 with stop_reason equal to refusal. Anthropic documents a companion stop_details object carrying a category and an explanation, and is explicit that it is populated only for refusals and can be null — so branch on stop_reason and treat stop_details as informational. Its full set of terminal reasons also includes end_turn, max_tokens, stop_sequence, tool_use and pause_turn, so a classifier can be exhaustive rather than a chain of substring tests.
• OpenAI-shaped Chat Completions. Two independent signals. A finish_reason of content_filter on the choice means output was omitted by safety filtering. Separately, the assistant message carries an optional refusal string, which is null when the model did not refuse and carries the refusal message when it did — and in that case content may be null, which is the crash above.
• A filtered deployment that raises. Some hosted deployments reject the request outright with an HTTP 400 whose body carries the filter result, so the decline arrives in your exception handler rather than in your response handler. If your classifier only runs on successful responses it will never see these at all.

One classifier, one enum

Handling three shapes at three call sites is how a deployment ends up with three subtly different definitions of a refusal. Collapse all of it into a single function whose output is a small closed set, and make every downstream decision read that enum rather than the response. Four values cover the ground: answered, refused on policy grounds, stopped for a capability or budget reason (truncation, a tool loop, a pause), and failed.

ANSWERED, REFUSED, STOPPED, FAILED = "answered", "refused", "stopped", "failed"

def classify_anthropic(resp):
    if resp.stop_reason == "refusal":
        return REFUSED, getattr(resp.stop_details, "category", None)
    if resp.stop_reason == "max_tokens":
        return STOPPED, "truncated"
    return ANSWERED, None

def classify_openai(resp):
    choice = resp.choices[0]
    if choice.message.refusal is not None:
        return REFUSED, "message.refusal"
    if choice.finish_reason == "content_filter":
        return REFUSED, "content_filter"
    if choice.finish_reason == "length":
        return STOPPED, "truncated"
    return ANSWERED, None

If you keep a phrase heuristic at all, keep it as a last-resort branch that runs only when the structural signals say “answered”, and attach a counter to it. That counter is the useful artefact: if it is near zero, the structural signals are covering your traffic and you can delete the heuristic; if it is high, you have found a decline shape your classifier does not know about, which is worth investigating rather than papering over.

One detail that catches streaming deployments: the terminal reason arrives on the final event of the stream, after text may already have been emitted. A decline can therefore follow partial output. Run the classifier on the terminal event, not on the accumulated text buffer, and make sure the consumer can retract or annotate what it already rendered.

A refusal is not a 429

The second half of the migration is the policy the fallback implements, and this is where teams accidentally build a loop. Retrying a policy decline against the same model with the same prompt will decline again; it is deterministic in a way a rate limit is not. So the retry table needs two columns, not one: what to do on a reliability failure (backoff, retry, fail over) and what to do on a decline (do not retry; serve the handled message, escalate, or route elsewhere as an explicit choice).

Routing a decline to a different model is a legitimate design, and it is a policy decision rather than an availability one — it should be logged as such, with the category recorded, so somebody can audit what you are re-serving and why. Anthropic ships a server-side fallbacks parameter that performs exactly this substitution inside a single call, behind a beta header; it is provider-specific and in beta at the time of writing, but the shape is worth knowing because it is the same shape you would otherwise build by hand.

Testing this without putting disallowed content in your repository is straightforward once the classifier is a pure function: record one fixture per decline shape — a Messages-API response with stop_reason set to refusal, a choice with a non-null refusal, a choice with content_filter, a 400 body — and assert the classifier’s enum output for each. That gives you a regression test that survives the next migration, which the phrase list never could. The wider retry design is covered in migrating workflow error recovery.