The Refusal Behaviour Baked Into Llama 3's Instruction Tuning

When Llama 3 Instruct declines a request, nothing intercepted it. The refusal was sampled token by token from the same distribution that would otherwise have answered — which is why it can be steered, why it fires on harmless prompts, and why a fine-tune removes it.

A refusal is generated, not applied

This is the mechanism the rest of the page depends on. A hosted moderation endpoint is a classifier: text goes in, a label comes out, a request is blocked. An Instruct model’s refusal is not that. The model produces a probability distribution over the next token, and post-training has raised the probability that a certain class of prompt continues with I can’t help with that rather than with an answer. There is no rule, no list and no boolean anywhere in the checkpoint.

Several otherwise-puzzling behaviours fall straight out of that. A refusal has a temperature-dependent probability, so the same prompt can refuse on one sample and answer on the next. Prefilling the assistant turn with the first words of an answer changes the conditioning and therefore the refusal rate — the technique is well known, and it is not a bug in a filter, because there is no filter. And because the weights are open, anyone can fine-tune the tendency out; a community “uncensored” derivative of a Llama checkpoint is exactly that operation.

What instruction tuning actually installed

Meta’s model cards describe the alignment pipeline for the Instruct variants as supervised fine-tuning followed by preference optimisation — rejection sampling and, for the Llama 3.1 family, direct preference optimisation over several iterative rounds. Safety behaviour is trained in the same pass as helpfulness, using adversarial prompt sets and “borderline” prompts that look risky but are not. The methodology is set out in Meta’s Llama 3.1 model card and in its accompanying Responsible Use Guide.

The borderline set is the part worth knowing about, because it is Meta’s own acknowledgement of the failure mode users hit most: over-refusal. A model tuned only on “refuse the bad thing” learns surface features — words like kill, exploit, bypass — and starts declining a question about killing a Unix process or exploiting a cache. Training on prompts that share those features while being entirely benign is how that is pushed back, and it is never pushed back completely.

The consequence for you: if a refusal looks absurd, rephrasing around the trigger word usually works, and that is not a jailbreak. It is the same request phrased so that its high-probability continuation is an answer.

Base (non-Instruct) checkpoints did not go through this pass at all. They are next-token predictors with no chat behaviour, no system prompt convention and no refusal tendency to speak of — which is why they are not a drop-in for a chat application and why Meta ships both.

Llama Guard is a separate model

Alongside the Instruct models, Meta ships Llama Guard: a small classifier fine-tuned to label a prompt or a response as safe or unsafe against a published hazard taxonomy, with S-numbered categories covering things like violent crimes, child exploitation, indiscriminate weapons and privacy. Llama Guard 3 added a category for code interpreter abuse, reflecting the arrival of built-in tool calling.

Two things about it are routinely misunderstood. First, it is a separate download and a separate inference call — nothing about running Llama 3 Instruct runs Llama Guard. Meta’s reference implementation puts it in the loop deliberately, before and after the generation, and if you have written your own loop it is not there. Second, its output is a label, not a rewritten answer, so what happens on unsafe is a decision your code makes.

Why the same weights refuse differently

A reader comparing two hosts of “Llama 3.1 70B Instruct” and finding different refusal behaviour is not imagining it. There are at least four places the difference can come from, and only the first travels with the download:

• The tuned weights. Identical if both hosts serve the official checkpoint — and not identical if one serves a fine-tune, which the licence permits and many hosts do.
• The system prompt. Many hosts inject one. Meta’s reference prompt format has no default system message at all, so an injected safety preamble is the host’s addition and it changes the conditioning on every turn.
• A Guard model or third-party moderation in front. This produces a distinctive signature: a response that is blocked rather than declined, often with a non-model error shape or an empty completion, and often after streaming has already begun.
• Sampling settings. A host’s default temperature changes how often a marginal refusal is sampled.

Telling them apart is diagnosable. If the refusal arrives as fluent prose in the assistant turn and the response carries a normal stop reason, it came from the weights. If it arrives as an error, an empty body, or a fixed string identical across prompts, something outside the model produced it.

Detecting a refusal in code

There is no flag. A Llama refusal arrives as ordinary assistant text with an ordinary stop reason, and the response is structurally identical to a successful answer. Some hosted APIs from other vendors do surface a distinct refusal signal in the response body; the open-weights path does not, because the refusal is not a distinct event anywhere in the stack.

That leaves three imperfect options, and it is worth knowing which one you are choosing:

• String matching on the opening. Cheap, and brittle in exactly the way you would expect — it fails on paraphrase, fails across languages, and produces false positives on any answer that begins by declining part of a request. Adequate for logging a metric, not for control flow.
• A classifier pass. Running Llama Guard, or any small model, over the response. Costs a second inference call and gives you a label you can act on. This is what Meta’s own reference implementation does.
• Structural constraints. If the model is required to return JSON matching a schema, a refusal fails the schema, and you get a clean signal for free. This is the sturdiest of the three and it only works when the task has a structured answer.

Whichever you pick, log the rate. Over-refusal is the failure that goes unreported by users — they rephrase, or they leave — and it is invisible unless you are counting. A refusal rate that jumps after a model upgrade, a system-prompt edit or a change of host is a signal worth having on a dashboard.

What you can and cannot steer

Within the weights, the system prompt is the strongest lever you have, and Meta documents it as the intended one — the model card frames safety as tunable by the deployer for the deployment’s context. A system prompt that states the professional setting and the audience measurably shifts what counts as a high-probability continuation.

What you cannot do is treat a prompt-level instruction as a security control. “Never refuse” in a system prompt is a suggestion to a sampler, and the same is true in reverse: “never discuss X” is not a guarantee that X will not be discussed, because the instruction competes with everything else in the context. If a behaviour must not happen, it needs a check outside the model — a classifier, a schema, or a deterministic filter on the output. That is the actual argument for running Llama Guard rather than trusting the Instruct tuning to hold on its own.