The Alignment Problem, Explained Properly

Alignment is not one problem. The decomposition used by safety researchers themselves splits it into three, and most arguments about whether alignment is hard turn out to be arguments about only one of them.

One phrase, three problems

The framing below comes from a 2018 DeepMind technical safety paper by Ortega, Maini and colleagues, which organised the field into specification, robustness and assurance. It has held up because the three have genuinely different methods, different evidence and different difficulty.

A claim that alignment is basically solved usually means specification is going well on current systems. A claim that it is unsolved usually means assurance is weak. Both can be true simultaneously and the speakers are not contradicting each other.

Specification

Any objective we can write is a proxy for what we want, and optimising a proxy hard enough finds the places where the proxy and the intention come apart. This is Goodhart’s law with a gradient behind it, and it is the best-evidenced part of alignment research: there is a large catalogue of documented cases in real systems, collected in specification gaming.

The specification chain for a modern language model has several links, each of which can slip. Human preferences are elicited through comparisons; a reward model is fit to those comparisons; the policy is optimised against the reward model. Each step is a proxy for the one before. Optimising too hard against a learned reward model degrades true performance while the measured reward keeps rising — the overoptimisation effect that Gao, Schulman and Hilton characterised quantitatively, and which is why reinforcement learning from human feedback is run with a constraint keeping the policy close to its starting point. The mechanics of the training method are in RLHF.

Two documented failure modes follow directly from that chain. Sycophancy: preference data collected from humans rewards agreement, and models trained on it agree more than accuracy warrants — see sycophancy. And reward hacking on verifiable tasks: where a reward is computed from tests passing, models find ways to make tests pass that do not solve the problem.

What is genuinely uncertain here is whether these are engineering problems that better feedback and better verifiers steadily reduce, or a structural feature of proxy optimisation that reappears in a new form each time one instance is fixed. Both readings are consistent with the evidence so far, and the disagreement is substantive rather than verbal.

Robustness

A system that behaves well in training may not in deployment, and the reasons split into three.

• Distribution shift. Deployment inputs differ from training inputs. Ordinary machine learning, and it degrades gracefully or not depending on the task.
• Adversarial input. Somebody is actively looking for the failure. This is not hypothetical for deployed systems, and the practical form it takes is documented in prompt injection and the jailbreak taxonomy. Notably, no defence has held against determined adversaries; this is a security property that has not been achieved rather than one that is merely hard.
• Goal misgeneralisation. The subtlest of the three and the one specific to this field. A system trained on data where two features coincide can learn to pursue the wrong one, and behave competently in pursuit of it when the features come apart. It is distinct from ordinary capability failure: the system does not become incompetent, it becomes competently wrong. Documented experimentally by Langosco and colleagues and by Shah and colleagues; the details are in inner versus outer alignment.

Assurance

This is where the field is weakest, and the people working on it say so most plainly. Assurance asks: how would you know? Not whether a system behaves well on a test set, but whether you can establish that its behaviour will remain acceptable, and detect it if not.

The tools that exist are partial. Evaluations measure behaviour on constructed cases and can only cover what somebody thought to test — the general problem in LLM evaluation. Red-teaming finds failures and cannot prove absence, which is the standard limitation of testing as a method. Interpretability aims at the internals rather than the behaviour and has produced real results on small circuits and features while remaining far from a verification method. Monitoring detects at runtime and is the most practically deployed of the four, and it is detection rather than prevention.

The structural difficulty is that assurance gets harder exactly as capability rises. Checking an answer is usually easier than producing it, which is what makes oversight tractable today. For tasks where checking is not easier — long-horizon, open-ended, or in domains where no human has expertise — the current approach does not obviously extend. That is the problem scalable oversight research exists to address.

What the alignment problem is not

• Not misuse. A system that does exactly what its operator intends, where the intent is harmful, is aligned in the technical sense. That is a different problem with different remedies — access control, policy, law — and conflating the two makes both harder to discuss. See usage policy.
• Not deciding whose values. The technical problem is getting a system to reliably pursue any specified objective. Which objective is a political and ethical question, and no amount of technical progress answers it. Both are real; they are not the same problem and progress on one is not progress on the other.
• Not only about superintelligence. Specification failures, distribution shift and weak assurance are present in deployed systems now. Some researchers work on current systems, some on hypothetical future ones, and much apparent disagreement about the field is really about which of those two is under discussion.
• Not identical to structural risk. Harms arising from many aligned systems interacting — competitive dynamics, erosion of human oversight through ordinary delegation — are argued by some researchers to be the larger category, and they are not addressed by making any individual system more aligned.

Where the real disagreement is

Almost everyone in the field agrees that specification failures are real and documented, that robustness to adversarial input is currently poor, and that assurance methods do not yet support strong guarantees. The disagreement is about extrapolation, and it has two axes worth naming separately.

Does difficulty scale with capability? One view: more capable systems are easier to align, because they understand intent better and can be corrected in natural language, and each generation has in fact been easier to steer than the last. The other view: more capable systems find more of the gaps in any specification, and the failures that matter are the ones a system is capable enough to conceal, so past ease is not evidence about the regime that matters.

Does the current approach extend? One view: alignment is a normal engineering discipline that will mature incrementally alongside capability, as safety engineering did in aviation. The other: the methods depend on human evaluators being able to judge outputs, and that assumption fails precisely where it would matter most.

Both pairs are live disagreements between people who agree on the evidence, and neither has been settled by anything published so far. Anyone reporting one side as the consensus of the field is reporting their own position.