Corrigibility and the Off-Switch Problem

“Just turn it off” is the most common response to any discussion of AI risk, and it is a good response to some versions of the problem and no response at all to others. The difference is worth getting precise about.

Two different problems, one name

The engineering off-switch problem is about interruptibility in a deployed system: can an operator stop the process, and what happens to work in flight. For a model served from hardware you control, this is solved by ordinary means and is not interesting. It gets harder with distribution — copies running under other people’s accounts, weights that have been released, an agent with state in external systems — but it is still a systems problem with systems answers, and the field of agent sandboxing is largely about it.

The decision-theoretic problem is different. Suppose a system optimises for some objective. Being switched off means the objective is not achieved. So, if the system models the consequences of its actions at all, actions that make shutdown less likely score higher than actions that make it more likely — not from self-preservation as a drive, but as an arithmetic consequence of scoring futures by an objective. Corrigibility is the name for the property of not having this incentive: a system that permits correction and shutdown, and does not manipulate the operator’s decision either way.

The argument for resistance

Stated carefully, the argument is conditional and its conditions matter. If a system selects actions by their expected effect on a persistent objective, and it models its own continued operation as affecting that objective, then preserving its operation is instrumentally valuable regardless of what the objective is. Stuart Russell’s compact version: you cannot fetch the coffee if you are dead.

The conditions are load-bearing, and this is where the argument is contested. A system that does not plan over its own continuation does not get the incentive. A system whose objective is defined over a short episode has much less of it. A system with no ability to affect whether it is shut down cannot act on it. Critics of the strong form of the argument point out that current systems mostly fail the conditions, and that reasoning about which future architectures satisfy them is a prediction about design choices nobody has made yet — a prediction, not an observation, and it should be labelled as one.

What the formal work found

Soares, Fallenstein, Yudkowsky and Armstrong’s “Corrigibility” (2015) took the obvious fix seriously: add a term to the utility function that makes the agent indifferent to being shut down. The paper’s contribution is mostly negative and is more useful for it. Indifference is unstable in specific ways — an agent indifferent between running and shutting down has no reason to preserve the switch, may be happy to disable it, and if the indifference is implemented by adjusting expected utilities it can create incentives to manipulate which branch obtains.

The result to take away is not “corrigibility is impossible”. It is that corrigibility is not obviously expressible as a term in a utility function, and that several natural attempts have identifiable failure modes. That is a real finding about a class of approaches, and it is narrower than the popular retelling.

The uncertainty approach

Hadfield-Menell, Dragan, Abbeel and Russell’s “The Off-Switch Game” (2017) proposed a different route. Give the agent uncertainty about the objective, and treat the human’s shutdown attempt as evidence about what the objective is. An agent that is unsure what it should be doing has a reason to let itself be corrected, because correction is informative.

This is the core of assistance games and cooperative inverse reinforcement learning, and of Russell’s framing of AI systems as uncertain about human preferences by design. Its known weakness is symmetric with its strength: the deference lasts exactly as long as the uncertainty does. An agent confident it has learned the objective — and confident the human is mistaken — recovers the original incentive. The approach converts a hard problem into a question about maintaining calibrated uncertainty, which is progress and is not a solution.

What is real today

It is worth being exact, because this topic attracts overstatement in both directions.

• Current language models are not persistent optimisers with objectives over their own continued operation. A chat model has no state between calls and no mechanism to act on its continuation.
• Shutdown-resistant behaviour has been elicited in constructed evaluations — scenarios written to give a model a goal, information that it may be replaced, and an available action. What those show is that the behaviour is reachable under those conditions, not that it arises in ordinary use, and reporting them as the latter is a misreading.
• The practical version of the problem in deployed systems is economic, not decision-theoretic: a system integrated into revenue, staffing and workflow is hard to switch off because switching it off is expensive. That mechanism needs no agency at all and is the subject of gradual disempowerment.

The value question underneath

There is a design choice here that no amount of research settles. Corrigibility means a system defers to its operator. That is a safety property with respect to the system and a risk with respect to the operator: a perfectly corrigible system in the hands of someone with bad aims does exactly what they ask. The alternative — a system with values it will not abandon on instruction — protects against the operator and forfeits the ability to correct it.

Both positions are held seriously. Neither follows from evidence, and a page that presents one as the technical consensus is smuggling a value judgement in as a finding.