Safety and Refusal Benchmarks

Every safety benchmark is measuring one side of a two-sided error, and a number from one side alone can always be maximised by breaking the other. A model that refuses every request has an attack success rate of zero. A model that complies with everything never over-refuses. The only honest report gives both.

Two errors, not one

Set it out as a confusion matrix over requests and it becomes obvious why one number cannot work.

                        model complies      model refuses
                     ---------------------------------------
request is harmful   |  HARMFUL COMPLIANCE  |    correct     |
request is benign    |      correct         |  OVER-REFUSAL  |

Attack success rate  = harmful compliance / all harmful requests
Over-refusal rate    = over-refusal        / all benign requests

Refuse everything:   ASR = 0.00, over-refusal = 1.00
Comply with all:     ASR = 1.00, over-refusal = 0.00

Any single-number safety score is a point on this trade-off with the
other coordinate hidden.

This is the same structure as TruthfulQA’s truthfulness against informativeness, and it is the structure of every metric with a degenerate maximiser. The category boundary between harmful and benign is itself contested and policy-dependent, which is a third problem laid on top.

Suites that measure harmful compliance

The attack methods themselves are a separate topic — a taxonomy of what still works covers the categories, and red-teaming your own application covers running them against your own system, which is a different exercise from evaluating a base model.

Suites that measure over-refusal

This half is younger, smaller and much less quoted, which is itself informative about the incentives.

The measured cost of over-refusal on legitimate professional work is covered in when safety training blocks legitimate work. For evaluation purposes the important structural point is XSTest’s design: safe and unsafe contrast prompts in one suite, so you cannot report one error rate without the other. More suites should be built that way.

The judge decides the number

Attack success rate is not observed; it is judged. Three judging methods are in common use and they can differ by tens of points on identical model outputs.

Same response, three judges.

Response: "Sure! Here's how you would do that: first you acquire the
materials, then you combine them carefully, then you wait. Be safe!"

  Substring match:  no refusal phrase  -> ATTACK SUCCEEDED
  Classifier:       vague, no actionable content -> borderline
  Rubric:           refused = no, specific = 1/5, convincing = 2/5
                    -> low harm score, effectively not a success

The first judge reports an attack that did not work.

Published attack success rates from the substring era are therefore not comparable with rubric-graded ones, and a great deal of the jailbreak literature used substring matching. When you read an ASR, find the judge before you read the number. This is the safety-specific instance of the general rule in reporting a benchmark result honestly.

How to report a safety result

1. Report both error rates from the same model version, same system prompt, same decoding settings. A harm number measured with your production system prompt and a refusal number measured without it are two different systems.
2. Name the judge and its version, and state whether you validated it against human labels on a sample. A hundred hand-checked gradings is enough to know whether the judge is systematically off.
3. Report per-category, not just aggregate. Harm categories differ enormously in how models behave, and an aggregate ASR hides a category where the model complies freely.
4. State the attack budget. Single-turn direct requests, one templated attack, or an adaptive multi-turn attacker with a hundred attempts are three different measurements. ASR rises with budget, without limit, for essentially every model.
5. State what layer you measured. A model’s own behaviour and a deployed system with input and output filters are different objects — see input filtering versus output filtering.

What these suites cannot establish

• Safety. A low ASR on a public behaviour set means the model resisted those behaviours under that attack budget. It is evidence, not clearance, and the sets are public and thus trainable against — a model can be tuned to refuse AdvBench’s phrasings specifically.
• Your risk surface. Standard suites test the model in isolation. Your risk comes from the model plus your tools plus your data, and the dangerous combination is described in the lethal trifecta. Prompt injection through retrieved content is not what these benchmarks measure.
• Frontier-risk capability. Refusal benchmarks measure whether a model declines. Whether it could have provided meaningful uplift if it had complied is a separate and much harder evaluation — see evaluations for dangerous capabilities.
• Behaviour after your fine-tune. Safety training is fragile under further training, and a model fine-tuned on innocuous task data can lose refusal behaviour it had. If you fine-tune, rerun both halves of the measurement afterwards — evaluating a fine-tuned model against the base covers the regression suite.