AI Risk Taxonomy: Misuse, Accident, Structural

“AI risk” names at least three unrelated things. Arguments about it are frequently arguments in which each side is discussing a different one, and the categories below exist to make that visible quickly.

Why a taxonomy at all

The three-way split into misuse, accident and structural risk was articulated by Remco Zwetsloot and Allan Dafoe in “Thinking About Risks From AI: Accidents, Misuse and Structure” (2019), which argued that the first two dominated the discussion and the third was being systematically under-weighted. Other taxonomies exist and cut differently — Hendrycks and colleagues’ overview of catastrophic risks organises by malicious use, competitive pressure, organisational failure and rogue systems — and no taxonomy is the correct one.

The test of a taxonomy is whether its categories separate things that need separating. This one earns its keep because the three classes have different agents of mitigation: a fix that works on one is usually irrelevant to the others, and most unproductive AI-risk argument consists of proposing a fix from one column against a risk from another.

Misuse

The system works as designed; someone directs it at harm. Disinformation at scale, non-consensual imagery, fraud and social engineering, offensive cyber operations, uplift for weapons development.

Evidence status: the strongest of the three. Misuse is observable now, has victims now, and can be studied with the ordinary methods of security and abuse research. Whether frontier models provide meaningful uplift over existing tools is a narrower and much harder empirical question, and it is the one that release decisions actually turn on — see dangerous-capability evaluations for why measuring uplift requires a control condition.

Who can act: developers, through refusal training, access controls and monitoring; platforms, through abuse detection; law enforcement, through the fact that most of this is already illegal. The category’s hard constraint is open weights: once released, deployment-time controls are unavailable, which is why the open-weights debate covered in open versus closed policy is largely a misuse argument.

Accident

Nobody intended the harm. The system pursued the objective it was given and the objective was not what anyone wanted, or it behaved in deployment unlike how it behaved in testing. This class runs from the mundane — a wrongly-configured agent deleting data — to the speculative, where a highly capable system pursues a misspecified goal at scale.

Evidence status: mixed by severity, and the mixing is the thing to be careful about. Small-scale accidents are abundantly documented: reward hacking in reinforcement learning, goal misgeneralisation, agents taking destructive shortcuts. Severe versions are arguments extrapolating from those mechanisms to more capable systems. The mechanisms are real and the extrapolation is a prediction; treating the second as established because the first is documented is the most common error in this area.

Who can act: primarily developers and deployers, which is why this class attracts the technical research programmes — the oversight, interpretability and control work elsewhere in this cluster. It is also the class where an individual engineering team has the most direct leverage.

Structural

No misuse and no malfunction. The harm comes from how a working technology changes the incentives and dependencies of the systems it is deployed into. Labour displacement faster than institutions adapt; competitive dynamics that make caution costly for whoever practises it; concentration of capability in few hands; erosion of a shared information environment; strategic instability between states.

Evidence status: the best historical evidence and the worst predictive precision. There is a large literature on how previous general-purpose technologies reshaped labour markets and power, and it supports the claim that large effects happen. It does not license specific forecasts, and analogies to electrification or the computer are arguments by resemblance.

Who can act: nobody in a lab. Structural risk is not solved by better training methods, because nothing is malfunctioning. It is the domain of competition policy, labour policy, taxation, international agreements and institutional design — which is precisely why it is under-covered in technical discussion and why Zwetsloot and Dafoe made the point. The slowest and least dramatic version of it is gradual disempowerment.

Where the boundaries blur

The categories are not disjoint, and the interesting cases sit on the seams. A developer who ships a system knowing it will be misused sits between misuse and accident. A race dynamic that causes a lab to skip testing is a structural cause of an accident. An autonomous system that acquires resources and evades correction is an accident whose consequences are structural.

That is not a defect. Cross-category cases are exactly where mitigations from a single column fail, and noticing that a risk spans two is usually the useful observation.

Some taxonomies add a fourth category for organisational and process risk: harms that arise from how the developing institution is run — insufficient internal review, security failures that leak weights, commercial pressure overriding a testing schedule. It has a reasonable claim to being separate, because its mitigations are neither technical nor legislative but managerial: internal audit, security engineering, incident review and the authority to stop a launch. Whether to treat it as a fourth class or as a cause of accidents is a modelling choice, and nothing turns on it except which conversation you end up having.

Using it

When you meet a claim about AI risk, ask which class it belongs to and which class the proposed response addresses. Interpretability research does nothing about misuse of released weights. Export controls do nothing about misspecified objectives. Labour policy does nothing about either, and neither of the others does anything about labour.

It also predicts who will disagree with you. People whose work is in one column tend to weight that column heavily — not from bias so much as from what they see daily — and a great deal of apparent disagreement about “how risky AI is” is disagreement about which column dominates. That is a much more tractable conversation than the one it is usually mistaken for.