P(doom): What the Number Means and Why It Varies

This page gives no number, including no range, and the reason is the content: a p(doom) is a compressed answer to at least four separate questions, and two people who state different figures are usually answering different questions rather than disagreeing about the world.

Where the notation came from

“P(doom)” began as informal shorthand in online AI-safety discussion — probability of doom, written as though it were a well-defined quantity in a model. It escaped into interviews and press coverage, where it is now routinely reported as a person’s position, in the way a poll result is reported.

The notation carries an implication the underlying claim cannot support: that there is a shared event whose probability people are estimating. There is not. The rest of this page is about what varies underneath.

The four hidden disagreements

1. What counts as doom

The word covers a wide range of outcomes that call for different responses and have different plausibilities:

• Human extinction.
• Permanent loss of human control over the future, with humans still present.
• A catastrophe with very large casualties from which civilisation recovers.
• Entrenchment of a small group’s power, made irreversible by capability asymmetry.
• “Loss of long-term potential” — a philosophical criterion that includes futures in which nothing visibly bad happens.

Someone using the first definition and someone using the last are answering different questions. Neither is misusing the term, because the term has no agreed definition.

2. By when

A probability with no time bound is not a probability of anything checkable. Common implicit horizons are “this century”, “within thirty years” and “ever, conditional on the technology being developed”, and the same person’s honest answer differs substantially between them.

3. Conditional on what

Unconditional, or conditional on AGI being built at all? On the current trajectory continuing, or after accounting for the regulatory and social responses that a visible approach to danger would provoke? A conditional estimate that assumes no one reacts is a different object from an all-things-considered forecast, and the two are frequently quoted side by side.

4. Doom caused by what

Misaligned autonomous systems, deliberate misuse, or slow structural displacement? These have almost nothing in common as causal stories and almost nothing in common as mitigations — see the risk taxonomy. A single number sums over them and discards the only information that would tell you what to do.

What kind of probability this is

It is a subjective credence about a unique, unprecedented, non-repeatable event. That is a legitimate use of probability under a Bayesian reading, and it has consequences worth being explicit about.

There is no reference class. Frequency estimates need a population of similar cases; the closest candidates — nuclear weapons, engineered pathogens, industrial transitions — are chosen by the person making the argument, and the choice of reference class largely determines the answer. That is not a criticism of anyone in particular; it is a structural feature of forecasting one-off events.

There is also no scoring. Forecasters get calibrated by making many predictions and being told which came true. On a question that resolves once, at the end, nobody’s credence has ever been scored. The general finding from the forecasting literature — associated with Philip Tetlock’s long-running work on expert political judgement — is that expertise in a domain does not by itself produce calibrated probability estimates about it, and that the feedback loop is what produces calibration. That loop is absent here for everyone.

Reading the surveys

The most-cited data comes from the AI Impacts surveys of researchers who published at major machine-learning conferences, run at intervals over the past decade — the most recent widely-discussed edition being the one reported as “Thousands of AI Authors on the Future of AI”. Rather than repeat a headline figure, here is what to check before treating any figure from it as evidence:

• Sampling frame. Authors at particular conferences are not a random sample of people with relevant expertise. They over-represent the subfields that publish there and under-represent safety-specific researchers, economists and security specialists.
• Response rate. These surveys are opt-in and the response rate is low. If interest in the topic correlates with answering, the sample is not the frame.
• Framing effects, which the surveys themselves documented. Asking about “high-level machine intelligence” and asking about “full automation of labour” produced materially different distributions from the same respondents. When wording moves the answer that much, the answer is partly a fact about the wording.
• What was actually asked. Several distinct questions about severe outcomes appear across these surveys, with different thresholds and horizons. Figures are frequently quoted without the question attached.
• Spread over centre. The informative result in these surveys is not the median. It is that the distribution is enormously wide — a large share of respondents give very low estimates and a meaningful share give high ones. That shape is the finding: this is a field without a consensus, and reporting a central tendency conceals exactly that.

What to do with the number

When you encounter one, the productive move is to ask for the decomposition rather than to argue about the figure. What outcome, by when, conditional on what, via what mechanism. In practice most apparent disagreements resolve substantially once those four are matched, and the residue — which is real — is a disagreement about specific mechanisms that can then be discussed on its merits.

It is also worth separating the empirical from the normative, because the two are usually stated in one breath. “The probability is non-trivial” is an empirical claim, however hard to check. “Therefore we should slow deployment” is a value and policy claim that additionally requires a view about the costs of slowing, about who bears them, and about how risk should be traded against benefit. People who agree about the first often disagree about the second, and no probability estimate settles it.

Finally: a single number is a poor decision input even when honest. Decisions turn on which risks are tractable, which mitigations are cheap, and which actions are robust across the disagreement — questions you can make progress on without anyone agreeing on a figure at all.