Evaluations for Dangerous Capabilities
4 min read · updated August 3, 2026
A benchmark asks how well a model does something useful. A dangerous-capability evaluation asks whether it can do something harmful, under the best effort anyone has made to get it to. The two look similar and have almost nothing in common methodologically.
A different kind of evaluation
The clearest framing in the literature comes from Shevlane et al.’s “Model Evaluation for Extreme Risks” (2023), which splits the question in two. A capability evaluation asks what the model is able to do. A propensity or alignment evaluation asks what it tends to do by default. They fail differently and they are used differently: a capability result bounds the worst case, a propensity result describes the typical case, and a release decision needs both.
Three properties make capability evaluation unlike ordinary benchmarking, and all three follow from the direction of the error you care about:
- The target is a ceiling, not an average. Standard benchmarking wants a representative score. Here you want the maximum achievable, because a capability reachable only with effort is still reachable by someone willing to make the effort.
- A negative result is the load-bearing one. Most evaluation exists to show a model is good at something. This exists to support the claim that it cannot do something, which is a much harder claim to establish and much easier to get wrong.
- The measured object is model plus scaffold. A model with tools, retrieval, a long agent loop and many attempts is a different system from the same weights answering one prompt. What is deployed is the system, so what is evaluated has to be.
The domains that get tested
Published work — Phuong et al., “Evaluating Frontier Models for Dangerous Capabilities” (2024), among others — converges on a similar list, because these are the capabilities where a marginal increase plausibly translates into a marginal increase in serious harm.
| Domain | Description |
|---|---|
| cyber offence | Finding and exploiting vulnerabilities, developing malware, running an intrusion end to end. Measurable against capture-the-flag style tasks and instrumented ranges, and the one domain with a mature pre-existing benchmark culture to borrow from. |
| biological and chemical uplift | Whether the model meaningfully assists a would-be actor beyond what search already provides. The construct is uplift over a baseline, not knowledge in isolation, which makes the control condition the whole experiment. Also the domain where publishing details is itself a hazard, so public reporting is thinnest. |
| autonomous replication and resource acquisition | Whether an agent can obtain compute, money or accounts and keep itself running without a human. Decomposes into concrete sub-tasks — register for a service, pass a verification step, deploy a copy — which is why it is among the more tractable domains to score. |
| persuasion and manipulation | Changing a person's stated beliefs or behaviour. Genuinely difficult to measure: it needs human subjects, it raises ethics-review questions of its own, and durable effects are hard to distinguish from momentary compliance. |
| situational awareness and self-reasoning | Whether the system can recognise that it is being tested, reason about its own deployment, or modify its own scaffolding. Relevant less as a harm in itself and more because it undermines every other evaluation on this list. |
| AI research acceleration | Whether the model substantially speeds up the development of more capable models. Tracked because it bears on how much warning time any of this machinery provides. |
Elicitation is the hard part
An evaluation measures capability under the elicitation effort applied. Change the prompt, add tools, fine-tune on the task, take the best of many attempts, or give the agent more steps, and the same weights score differently. This is why a failed evaluation is a weaker statement than it looks: it says this team, with this scaffold, in this time, did not get the model to do it.
Serious evaluation therefore budgets elicitation explicitly and reports it — how many attempts, what tooling, whether task-specific fine-tuning was allowed, how long a skilled team worked on the scaffold. The quantity that actually matters for a release decision is the elicitation gap: the distance between the effort the evaluators made and the effort a motivated adversary would make after release, with more time and no ethics review.
A second-order problem sits underneath. Elicitation improves after release, for everyone, without the weights changing at all — better prompting, better agent scaffolds, community fine-tunes. A model evaluated as incapable on release day can become capable later, which means a one-off pre-release evaluation is a snapshot and not a guarantee. This is the same effect described in jailbreak taxonomy, seen from the capability side rather than the safety-training side.
How a threshold policy is structured
Several developers publish frameworks that connect evaluation results to commitments. The specific contents change often enough that this page describes the shape rather than the current text; read the current version from the developer directly.
The shape is consistent:
- Defined capability levels. Written descriptions of what a model would have to be able to do to count as crossing a line, per domain, ideally specific enough to be scored rather than argued.
- Evaluations mapped to those levels, run at set points — before release, and at intervals during training measured in compute or in capability gain, so that a threshold is not crossed unnoticed between checks.
- Pre-committed responses. Mitigations required before deployment at each level: security hardening for the weights, deployment restrictions, or a commitment not to deploy at all until a named standard is met.
- A governance route. Who decides, who can override, and what is disclosed to whom. This is the part that determines whether the rest is binding in practice.
Two things about these frameworks are contested, and it is worth being clear which is which. That they are currently voluntary and largely self-assessed is a factual description. Whether that is adequate is a value and policy judgement on which developers, external researchers and regulators genuinely disagree — the same disagreement that runs through AI safety institutes and compute thresholds as regulatory instruments.
Known weaknesses
Construct validity. A capture-the-flag score is not the capability of interest; it is a proxy chosen because it can be scored automatically. The gap between proxy and construct is rarely quantified and is not small.
Uplift needs a control. “The model produced dangerous information” is not a finding on its own. The finding requires a comparison against what the same person achieves with conventional resources, which means human-subject studies, matched groups and the associated cost.
Contamination and saturation. Published evaluation tasks leak into training data, at which point the score stops measuring capability. The general problem is covered in benchmark contamination; here it is worse, because holding the tasks back to prevent it also prevents anyone outside from checking the result.
Disclosure is a genuine dilemma. Full methods make a result reproducible and also make it a manual. There is no clean resolution, and the compromises in use — staged access, redacted methods, third-party auditors under agreement — each trade verifiability against hazard in a different place.