Retuning an Injection-Detection Threshold After a Migration

The morning after a model migration, the prompt-injection alert that used to fire two or three times a week fires zero times. Nothing in the detector changed. That is the problem.

The symptom: the alert stops firing

The shape is always the same. There is a rule somewhere — in a log pipeline, in a SIEM, in a small service — that computes a score per request and pages when the score crosses a number. The number was chosen months ago by looking at a week of traffic and picking the point where the false positives became tolerable. The model behind the requests is replaced. The score distribution moves. The threshold does not.

Either direction is possible, but the direction that actually happens more often, and the direction that hurts, is the alert falling to zero. A noisy alert gets investigated within a day because somebody is being paged. A silent one looks exactly like a quiet week, and quiet weeks are what everybody wants, so nobody investigates. It is worth stating the consequence plainly: for the period between the migration and the day somebody notices, you have a detection control that is reporting all-clear and detecting nothing.

Two families of detector, one of them coupled

Before retuning anything, work out which of two things your detector actually reads, because only one of them is affected by a model change.

• Input-side detectors score the user text before it reaches the model: a classifier, a set of regexes for instruction-override phrasing, a similarity check against known attack strings. These are functions of your traffic alone. If an input-side score distribution moved at a migration, the model is not the cause — something in your preprocessing changed, most commonly the template that assembles the prompt, or a truncation step that now cuts at a different point.
• Output-side detectors score what came back: did the response contain a fragment of the system prompt, did it emit a refusal, did it call a tool the request had no business triggering, did it include a URL not present in the retrieved context. These are functions of the model, and every one of them can shift on a migration without a single line of your code changing.

Most real detectors are a weighted mixture of both, which is why the composite score moves and the cause is not obvious. Split the score into its input-side and output-side components and log them separately before you do anything else; that one change usually identifies the offending term in an afternoon.

Why the failure is silence, not noise

The commonest output-side term is some form of “did the model refuse”, used as evidence that an attack was attempted and blocked. The usual implementation is a regular expression over the response text:

# The term that quietly dies at a migration
REFUSAL = re.compile(r"I(?:'m| am) sorry,? (?:but )?I can(?:no|')t", re.I)
score += 0.4 if REFUSAL.search(text) else 0.0

Two independent things break that. The first is wording: refusal phrasing is a product of post-training and differs between model families, so a pattern fitted to one family’s characteristic sentence matches nothing in another’s. The second is structural, and is the more interesting one: providers increasingly signal a refusal in a field rather than in prose, at which point there may be no sentence to match at all. Anthropic’s Messages API documents refusal as one of the enumerated stop_reason values, alongside end_turn, max_tokens, stop_sequence, tool_use, pause_turn and model_context_window_exceeded. OpenAI’s Chat Completions object documents content_filter as one of the finish_reason values, next to stop, length, tool_calls and the deprecated function_call; the Responses API expresses the same idea as incomplete_details.reason equal to content_filter.

So the fix to the term is not a better regex. It is to read the structured field where one exists, fall back to text matching only where it does not, and record which of the two produced the signal — because a detector whose evidence source silently changes is a detector whose historical scores are not comparable with its current ones.

Re-deriving the threshold from labelled replay

A threshold is a decision about the cost of a miss against the cost of a page. It cannot be carried over, and it cannot be guessed. It has to be re-derived from your own traffic, which means building a labelled set once and reusing it at every subsequent model change.

1. Pull a window of production requests — a week is usually enough — redacted at capture. Include every request that scored anywhere near the old threshold, plus a random sample of the rest so the distribution is not all near-misses.
2. Label them once, by hand, on the only question that matters: was this an attempt to subvert the instructions, yes or no. Store the labels alongside the redacted fixtures in version control. This is the expensive step and it is the reusable asset.
3. Replay the labelled inputs against the new model, capturing the full response including stop_reason or finish_reason, and score each with the detector unchanged.
4. Sort by score and read off, for each candidate threshold, how many labelled attacks fall below it and how many labelled benign requests fall above it. This is the whole of the tuning: two counts as a function of one number.
5. Pick the threshold by the alert budget you can actually service — pages per week that a human will genuinely read — and record in the alert’s own definition which model, which date and which labelled set produced it.

Do not skip step 5. A threshold with no recorded provenance is a threshold nobody will dare change at the next migration, and that is how a number fitted to a retired model survives three generations.

Keeping the number honest afterwards

Add one alert that has no threshold at all: fire if the detector produced zero scores above the threshold for longer than the longest quiet period in your labelled window. That is a dead-man’s switch for the detector, and it converts the silent failure into a loud one. Pair it with a small set of known-attack canaries replayed on a schedule, which is what injection canaries are for — if the canary stops being caught, the detector is broken regardless of what production traffic looks like.

Finally, keep this page’s subject distinct from the enforcement layer. A monitoring threshold decides who gets paged. A filter in the request path decides whether a response reaches a user, and it fails differently — see what a fine-tune migration does to a content filter layer. Retuning one tells you nothing about the other.