Constitutional AI and RLAIF

Constitutional AI is often described as “giving the model rules”. It is not that. It is a way of generating a training signal without a human in the loop for every example, using a written document as the thing the signal is derived from.

The problem it was built for

Reinforcement learning from human feedback, described in RLHF explained, needs people to compare model outputs. That has three costs which get worse as models improve. It is expensive per label. It exposes annotators to the worst content the model can produce. And the resulting policy is implicit — it lives in a reward model fitted to a pile of preferences, so nobody can read what was actually taught, and adjusting it means collecting more labels.

Constitutional AI, introduced by Bai et al. (2022) at Anthropic, moves the human contribution from labelling individual outputs to writing the principles that decide them, and lets the model apply those principles at scale. The generic name for using model-generated preferences in place of human ones is RLAIF — reinforcement learning from AI feedback. Constitutional AI is one particular way of doing it.

The two phases

Phase one: supervised, by self-critique

Prompt the model, including with prompts designed to elicit harmful answers. Take its response and ask it, in a separate call, to critique that response against one principle drawn from the constitution. Then ask it to rewrite the response in light of its own critique. Keep the revision, discard the rest, and fine-tune the model on the collected revisions.

prompt          -> initial response
+ principle p   -> critique of that response
+ critique      -> revised response
                   ^ keep (prompt, revised) as SFT data

The output of this phase is a model whose default behaviour has already moved, before any reinforcement learning has happened.

Phase two: preferences, generated by a model

Sample pairs of responses from the phase-one model. Ask a model — given a principle — which of the two better satisfies it. Those comparisons train a preference model exactly as human comparisons would, and the policy is then optimised against it with the usual machinery described in DPO versus PPO. The substitution is narrow and worth stating precisely: the algorithm is unchanged, and only the source of the comparison labels differs.

What a constitution actually is

A short list of natural-language principles, each usable as an instruction in a critique or a comparison prompt. Not code, not a formal specification, not a filter applied at inference time. Published examples draw on human-rights instruments, platform policy language and purpose-written clauses about honesty and harm.

Two properties follow from it being a document. It is legible: anyone can read what the model was trained toward and disagree with a specific clause rather than with a vibe. And it is editable: changing the intended behaviour is a text edit followed by a training run, not a new annotation campaign. Whether the trained model actually conforms to the edited text is a separate empirical question, and one the method does not answer by itself.

What it buys

• Scale. Preference data becomes a compute cost rather than a hiring problem, which is the reason the technique spread.
• Reduced annotator exposure. Humans need not read every harmful generation to label it.
• An auditable target. The intended policy exists as text. This is a genuine transparency gain over a reward model, and it is the property that most distinguishes the approach.
• Better refusal behaviour, on the paper’s own terms. The original work reported models that decline harmful requests while explaining why, rather than refusing flatly — a direction relevant to false refusals.

What it relocates rather than solves

Who writes it. The hard question in any alignment scheme is whose values, and a constitution makes that question visible without answering it. Making the target legible is a real improvement over leaving it implicit; it is not agreement about the target. That is a value question, and no training method settles it.

The evaluator is the thing being evaluated. Phase two asks a model to judge outputs from a model of similar capability. Any systematic misreading of a principle is applied consistently across the whole preference dataset, which is a different and quieter failure than noisy human labels — the same concern as judge bias in evaluation, with higher stakes because it is baked into weights. Whether this degrades as the judged model approaches or exceeds the judge is exactly the question scalable oversight exists to study.

Text does not compel behaviour. The constitution shapes a training signal; the model learns whatever fits that signal, which may be a narrower or wider rule than the clause states. There is no step in the pipeline that verifies the learned policy matches the written one, and no current method that would verify it in general.

The comparison to human feedback is not settled. Whether AI-generated preferences match or beat human ones is an empirical question whose answer depends on the judge model, the domain and the principle being applied — a judge is reliable on clear-cut cases and unreliable exactly where the principles conflict, which is where the interesting examples live. Reported comparisons are specific to the setup that produced them and do not generalise into a claim that AI feedback is better or worse in general. In practice, production pipelines tend to be hybrids: model-generated preferences for volume, human labels for the cases where the principles collide or the stakes are highest.

It addresses outer alignment. The technique is about specifying what to train toward. It says nothing about whether the learned system pursues that objective for the reasons intended, which is the separate problem covered under inner alignment. A page that presents constitutional training as a solution to alignment in general is overstating it, and the original paper does not.