The Documented Safety Training Behind Phi-3's Refusals

When Phi-3 declines a request, nothing inspected that request. The refusal is the highest-probability continuation, produced by the same next-token machinery as any other sentence, because of a training process Microsoft documented in detail.

What Microsoft published

There are two primary sources, and they cover different halves. The Phi-3 Technical Report, published in April 2024, describes the post-training stack: supervised instruction fine-tuning followed by direct preference optimisation on preference data covering helpfulness and harmlessness, together with evaluation against responsible-AI benchmarks for groundedness, harmful-content continuation and jailbreak resistance.

The methodology itself was then published separately as “Phi-3 Safety Post-Training: Aligning Language Models with a ‘Break-Fix’ Cycle” in July 2024. That paper is the one to read if you want to know why the model refuses what it refuses, because it describes the loop that produced the refusal behaviour rather than just asserting that alignment happened.

The break-fix cycle

The published process is iterative rather than a single alignment pass. Each round runs the same four stages:

1. Dataset curation. Assemble preference data for the harm categories in scope, mixing publicly available datasets with data generated in-house.
2. Safety post-training. Apply supervised fine-tuning and preference optimisation using that data, adjusting the model’s distribution towards refusal on the targeted categories and away from over-refusal on benign neighbours.
3. Red teaming. Have the Microsoft AI Red Team attack the resulting checkpoint, looking for prompts that elicit the behaviour the previous stage was supposed to remove.
4. Evaluation and vulnerability identification. Score against responsible-AI benchmarks, characterise what broke, and feed it into the next round’s dataset.

The paper reports multiple rounds of this loop, with the failures found by each round’s red teaming becoming the training data for the next. That is the “break” and the “fix”.

The structure tells you something the headline claim does not: the refusal boundary is defined by what the red team happened to find, across a finite number of rounds, in the harm categories chosen at the start. It is an empirical boundary, not a specification. Nothing about the process produces a guarantee, and the paper does not claim one.

A refusal is a sampled continuation

This is the part that matters operationally. In a hosted API, a refusal may come from a classifier sitting in front of or behind the model — a separate system that can reject a request before the model ever sees it, and that often surfaces as a distinct response shape or error. Phi-3 downloaded from Hugging Face has no such component. Preference optimisation changed the weights; there is nothing else in the box.

Several observable behaviours follow directly:

• Refusals have no distinct signal. There is no field to check. A refusal is text, and detecting one means matching on phrasing — which is fragile in exactly the way string-matching on model output always is.
• Sampling parameters affect them. A refusal is a probability, not a switch. Raising temperature raises the chance of drawing a non-refusing continuation on a borderline prompt, which is an argument for temperature 0 on anything where consistency of policy matters.
• The system turn shifts the boundary. Because the behaviour was tuned in the presence of a system message, the content of that message moves the distribution. A permissive system prompt genuinely changes what the model will do, which is a feature when you own the deployment and a hazard when the system turn is assembled from user input.
• Prefilling defeats it. The template is text, so ending the prompt after <|assistant|> with the first words of a compliant answer makes refusal a far less likely continuation. See the chat template page for how that is constructed. Anyone who can reach your raw prompt interface can do this.

Why small models refuse imprecisely

Alignment training has to draw a boundary through a space of requests, and drawing it requires the model to represent the distinction between “how do household chemicals react” asked by a curious person and the same question as a step in something harmful. A 3.8B-parameter model has less capacity to hold that distinction than a frontier model does, and the boundary it learns is correspondingly blunter.

The visible symptom is error in both directions on the same checkpoint: false refusals on benign prompts that merely resemble a trained category — security research, medical questions, fiction involving conflict — alongside compliance on genuinely unwanted requests that are phrased outside the shapes the red team explored. Tightening the training to reduce one increases the other. This is a capacity constraint, not a tuning oversight, and it is the reason the break-fix paper spends attention on over-refusal as a tracked metric rather than treating refusal as strictly good.

Detecting a refusal in practice

You will need to know when it happened — to retry, to fall back, to show a different interface, or simply to count. Three approaches, in increasing order of how much they cost and how well they work.

String matching on phrases like “I can’t” or “I’m unable to” is what everybody writes first. It is fast, it is free, and it is wrong in both directions: it misses refusals phrased differently, and it fires on an answer that begins by explaining what the model cannot verify before answering anyway. Treat it as a signal for triage, not as a decision.

Asking the model to signal works better because it uses the structure you already control. Require a JSON envelope with a status field, constrain it with a schema, and read the field rather than the prose. This makes refusal machine-readable and testable, which is the thing string matching cannot give you at any price. It does not make refusals more accurate — the same blunt boundary decides the field’s value.

Reading the logits is the mechanism-level version. Because the refusal is a continuation, the model’s uncertainty about whether to refuse is visible in the distribution over the first few tokens of the answer. A generation where the top two candidates are a refusal opener and a compliant opener, separated by very little, is a borderline case by the model’s own reckoning. Logging that gap gives you a calibration signal that no amount of output parsing will, and it costs one extra field in the generation call.

Whichever you use, log the rate. A refusal rate that moves after a prompt change, a checkpoint bump or a temperature adjustment is telling you something about your system, and you cannot see the move if you were not counting before it.

What this leaves you responsible for

Phi-3 ships under MIT, which as the licence page sets out imposes no acceptable-use policy at all. So there is no contractual constraint on your deployment, no provider-side filter, and a model-side boundary that is empirical and blunt. Everything between that and your users is yours to build.

In practice that means a separate moderation stage rather than a reliance on the weights: classify input before it reaches the model and output before it reaches the user, using something built for the job — a dedicated moderation model such as those covered in the Llama Guard page — and log what it catches. A refusal you cannot detect is also a refusal you cannot audit, and if you cannot audit it you cannot tell a regulator, a customer or yourself what your system actually does.