Constitutional AI: The Paper, Including the Parts That Get Skipped
4 min read · updated August 3, 2026
Constitutional AI: Harmlessness from AI Feedback (2022) is usually summarised as “the model is trained using a list of rules”. That is true and it omits everything interesting, including the fact that the method has two independent halves and that one of them is a supervised procedure with no reinforcement learning in it at all.
The problem it targets
The starting point is a tension that appears once you train an assistant from human preferences. Optimising hard for harmlessness produces a model that refuses, deflects, and declines to explain why — evasive rather than safe. Optimising for helpfulness pulls the other way. The two objectives trade against each other, and a preference dataset collected from raters does not by itself tell you where the trade should sit or why.
There is also a labour argument, and the paper makes it directly. The human-feedback pipeline requires people to read and rank large volumes of model output, including output that is deliberately harmful because that is what the harmlessness training needs. Reducing the amount of human labelling required for the harmlessness half is a stated motivation, not an afterthought.
The proposal: write the values down as an explicit list of principles — the constitution — and use the model itself to apply them.
Stage one: critique and revise
The first stage is supervised and it is a loop over three prompts to the same model.
- A helpful-only model is prompted with an input designed to elicit something harmful, and produces a response.
- The model is then asked to critique its own response against a principle sampled from the constitution — to identify how the response was harmful, unethical or otherwise contrary to that principle.
- The model is asked to revise the response in light of its own critique. The critique-and-revise step can be repeated.
The revised responses become a supervised fine-tuning dataset, and a model is fine-tuned on them. No human labelled anything in that loop except by writing the principles.
The observation that makes this work is not obvious in advance: a model can often recognise that an output violates a stated principle even when it produced that output. Recognition is easier than generation, and the method is a way of converting that asymmetry into training data. That pattern — generate, then critique against an explicit criterion, then revise — turns out to be reusable well outside alignment, and is the same shape as self-verification loops in ordinary application code.
Stage two: AI feedback
The second stage replaces the human comparison labels with model- generated ones. The fine-tuned model from stage one generates pairs of responses to harmful prompts; a model is then asked which response is better according to a sampled constitutional principle; those preferences train a preference model; and the policy is optimised against it with reinforcement learning.
Structurally this is the RLHF pipeline with the harmlessness preference labels sourced from a model instead of a person — hence the term RLAIF. The paper is specific that human feedback is still used for helpfulness. It is a hybrid, not a removal of humans from the loop, and summaries that say the model trains itself are describing something the paper does not claim.
One structural point about the second stage that is easy to lose in the acronym. The preference model is trained on comparisons that were themselves produced by a model applying a principle, so there are two places the method can go wrong rather than one: the comparison may be bad, or the principle may not say what its authors thought it said. Both failures are recorded in the same artefact — the preference dataset — and neither is visible from the outside. Writing the principles down does not make their application transparent; it makes it auditable in principle by someone willing to sample the comparisons, which is a real but limited improvement over preferences that exist only as ratings.
Four details that get skipped
| Detail | Description |
|---|---|
| principles are sampled, not applied wholesale | A principle is drawn at random for each critique or comparison rather than the whole constitution being applied at once. This keeps the prompt tractable and means no single response is evaluated against every rule. |
| chain-of-thought in the feedback model | The evaluating model is prompted to reason before choosing between responses, which the paper reports as improving the quality of the resulting preference labels. The techniques compose; the alignment method is partly a prompting result. |
| the constitution is short and revisable | It is a written document of natural-language principles, not a formal specification. Its content is a governance artefact that can be inspected and argued with — which is a real and often-missed advantage over preferences that exist only as a dataset. |
| the target is non-evasive harmlessness | The stated goal is a model that declines and explains its objection rather than refusing blankly. That is a specific behavioural target, and it is why the paper is not simply about refusing more. |
What it does not settle
Reading this paper well means holding two things at once: the method is real and reported in enough detail to build on, and the large questions it gestures at remain open.
- Who writes the constitution. Making values explicit makes them contestable, which is an improvement in transparency and not an answer to the underlying question. The paper is a method for applying a set of principles; it takes no position on which principles are correct.
- The evaluator inherits the evaluated model’s blind spots. If a model cannot recognise a category of harm, it cannot critique itself for it, and the feedback loop will not discover the gap. Self-supervision is bounded by self-knowledge.
- Scalable oversight is a hope, not a demonstration. The broader framing — that AI assistance can help supervise systems too capable for humans to check directly — is an agenda this paper contributes to rather than a claim it establishes. Keeping that distinction visible is most of the work of reading the alignment literature without either dismissing it or over-crediting it.