Gradual Disempowerment vs Sudden Takeover

Most public discussion of losing control to AI imagines a moment: a system becomes capable, acts, and the situation changes. A second scenario has no moment at all, and several researchers consider it the more likely of the two.

Two stories

Both are predictions. Neither is a finding, and this page compares them on the assumptions each requires rather than asserting which is right, because the evidence available now does not settle it.

The takeover story and what it needs

The strongest version, associated with Bostrom’s Superintelligence and with earlier work by I. J. Good, runs: capability improves, possibly sharply if a system contributes to its own improvement; a system pursuing a misspecified objective finds that resources and freedom from interruption serve almost any objective; and a sufficient capability gap means countermeasures fail.

It requires several things to hold together. A large capability gap opening faster than institutions notice. A system with persistent goals that plans over its own continuation. Enough of a decisive advantage that the rest of the world’s response does not matter. Critics attack each of these, most often the compression of the timeline: the argument frequently assumes the transition is fast enough that nobody reacts, and that assumption is doing more work than the capability claims.

It has one clear advantage as a story: it identifies a thing to prevent, and it is the reason most technical safety work exists.

The gradual story and what it needs

Paul Christiano’s essay “What failure looks like” (2019) described a version in which optimisation for measurable proxies drifts away from what anyone wanted, across a whole economy, with no single point of failure. Kulveit and colleagues developed the argument at length in “Gradual Disempowerment” (2025).

The core observation is about dependence. Human influence over large social systems today rests substantially on those systems needing humans. Firms need workers and customers. States need soldiers, taxpayers and some measure of consent. Culture is produced and consumed by people. Each of these is a feedback channel through which human interests get represented, and each operates whether or not anyone intends it to.

If AI systems can perform the labour, produce the content, and generate the value that these processes require, the dependence weakens. Not by anyone’s decision — by many separate, locally sensible choices to use a cheaper input.

The mechanism, stated carefully

The nearest well-studied analogue is the political-economy literature on resource-dependent states: where public revenue comes from extraction rather than from taxing citizens’ work, the incentive to be responsive to citizens weakens. Researchers cite this as an illustration of a mechanism — funding independent of the population reduces accountability to it — and not as a prediction that AI economies resemble oil economies in other respects. The analogy is suggestive; it is not evidence about AI.

Three features distinguish this from ordinary automation worries:

• No decision point. There is no moment at which someone chooses to reduce human influence, so there is no moment at which a decision could be reversed. Previous automation waves displaced tasks while leaving humans as the ultimate consumers, voters and soldiers; the claim here concerns those roles too.
• Each step is locally rational. Every substitution makes sense for the organisation making it, which is why the trend does not depend on anyone being careless or malicious.
• Competitive lock-in. An organisation that keeps humans in the loop for reasons of principle competes against ones that do not. Whether that pressure dominates in any specific sector is an empirical question with real variation.

Objections to the gradual story

It is hard to falsify. A prediction of slow drift with no discrete event is compatible with many observations, and a claim compatible with everything is evidence of little. Proponents respond by naming indicators — the share of economic value produced without human labour, the degree to which policy responds to public preference — but these are noisy and contested measures.

It underrates political agency. Societies have repeatedly regulated technologies that damaged widely-shared interests, sometimes slowly and partially, but they have done it. The gradual story tends to model institutions as passive.

The economics are contested. Whether AI substitutes for human labour broadly or complements it, and over what horizon, is an open question in economics with serious researchers on multiple sides — the subject of post-AGI economics and of the existing work on AI and employment data. The disempowerment argument assumes broad substitution.

Ordinary redistribution may cover it. If the problem is that people stop earning, the standard toolkit of taxation and transfers exists — see UBI and automation. The counter-argument is that transfers address income and not influence, and that a population dependent on transfers has less leverage than one whose labour is needed. That is a claim about political economy, and it is disputed.

Comparing them honestly

The gradual scenario needs fewer unusual assumptions. It does not require a capability discontinuity, a system with persistent goals, or any deception. It requires that AI substitutes for human labour widely enough, and that existing institutional feedbacks depend on that labour more than on anything else. Both are contestable, and both are ordinary empirical questions rather than exotic ones.

The takeover scenario needs stronger assumptions and, if they hold, is worse and faster. It is also more tractable to prepare for, because it names a specific thing to detect and prevent. Which is why the coverage asymmetry runs the way it does: the sudden story has a villain, a moment and a countermeasure, and the gradual one has none of those and no natural point at which anyone is supposed to act. That is a fact about what makes a compelling narrative, not evidence about which is more likely.