RLAIF: The Model as Preference Labeller
10 min read · updated August 4, 2026
RLAIF replaces the human who chooses between two responses with a model that chooses between two responses. Everything downstream — the preference loss, the reward model, the policy update — is unchanged. The entire question is whether the labels are good enough, and in what way they are wrong.
The swap, and what stays the same
In RLHF, the pipeline is: collect pairs of responses, have humans choose, train a reward model on those choices, optimise a policy against the reward model. RLAIF changes exactly one step. A capable model is shown the prompt, both responses and a written instruction about what counts as better, and it chooses.
The motivation is straightforward arithmetic. Human preference collection is slow, costs money per comparison, and its throughput bounds how much data any post-training run can use. A model labeller runs at inference cost and can label overnight what an annotation team would take months to produce. It also produces a written justification for free, which turns out to matter for auditing.
What does not change is important. The scale is still unidentified, the reward model still over-optimises, and the policy still hunts for the highest-scoring region of the reward surface. RLAIF changes where the preferences come from, not any of the machinery that makes preferences dangerous to optimise against.
Constitutional AI, the first published form
The method with the clearest published description is Constitutional AI, Bai et al. (2022, arXiv 2212.08073). It has two phases and the second is where the AI feedback appears.
- Supervised phase. The model generates a response, then critiques its own response against a written principle from a short list — the “constitution” — and then revises it. The revised responses become supervised fine-tuning data. No human labels are involved beyond writing the principles.
- Feedback phase. The fine-tuned model produces pairs of responses. A model is asked which of the pair better satisfies a principle drawn from the constitution. Those choices train a preference model, which then drives reinforcement learning in the usual way.
The structural claim worth taking from this is that the human input moves rather than disappearing. Instead of thousands of individual comparisons, a human writes a small number of explicit principles, and those principles are inspectable, versionable and arguable in a way a pile of pairwise choices is not. Whether that is an improvement depends on whether the principles capture what the annotators would have done, which is an empirical question and not a settled one. The wider treatment is on Constitutional AI.
What the comparisons actually claim
The most directly relevant comparison is Lee et al. (2023, arXiv 2309.00267), which trained policies from human feedback and from AI feedback on the same tasks and had humans rate the results. On summarisation, human raters preferred both the RLAIF and the RLHF policies over the supervised baseline at broadly similar rates. The paper also examined using an off-the-shelf model’s scores directly as the reward, skipping the separately-trained reward model.
There is a second, less contested argument for AI feedback that does not depend on matching humans at all: consistency. A model applies the same criterion to the ten-thousandth pair as to the first. A human annotation pool has turnover, drift, fatigue and differing interpretations of the guidelines. For any criterion that can be written down precisely, the model is more reliable at applying it — which says nothing about whether the criterion is the right one.
The bias a single labeller concentrates
This is the mechanism that matters most and it is rarely stated plainly. Human annotators make mistakes that are largely independent of each other. Person A misreads a question, person B is tired, person C has an unusual preference about tone. Aggregate over thousands of annotators and the independent errors partly cancel; what survives is the shared signal.
A single labeller model makes the same mistake every time. Its errors are perfectly correlated across the dataset, so they do not cancel — they accumulate into a consistent tilt in the reward function. The policy then optimises against that tilt with full optimisation pressure, and it will find it, because a systematic bias is exactly the kind of structure gradient descent is good at exploiting.
The practical consequence is that the failure mode changes character. Human labels give a noisy reward model. AI labels give a precise reward model that is confidently wrong in one direction, and confident wrongness is much harder to notice in aggregate metrics — the variance looks better, which reads as an improvement.
Four biases documented in LLM judges
| Bias | Description |
|---|---|
| position | Which response is presented first affects which is chosen, independently of content. The standard mitigation is to evaluate each pair in both orders and keep only pairs where the judgement agrees, which also gives you a free estimate of how large the effect is on your data. |
| length | Longer responses are favoured, the same effect documented for human preference data by Singhal et al. (2023). Since the policy will amplify whatever the labeller prefers, an untreated length bias in the labeller becomes a verbose model. |
| self-preference | Models tend to rate their own generations more highly than equally good text from other models; Panickssery et al. (2024) examined this directly. It is a specific hazard when the labeller and the policy are the same model, which is the cheapest and therefore most common configuration. |
| style over substance | Confident tone, structure and formatting are easier to assess than factual correctness, and judges weight them accordingly. A response that is well-organised and wrong can beat one that is messy and right. |
Every one of these is measurable on your own data before you use a labeller for anything. Position bias in particular takes one afternoon: label a few hundred pairs in both orders and count the disagreements. If a third of your labels flip when you swap the order, a third of your preference data is noise with a direction.
How to use it without inheriting all of that
- Keep a human-labelled evaluation set the labeller never touches. This is the only thing that can tell you the AI labels have drifted from what you meant. A few hundred carefully labelled pairs is enough and it is the highest-value annotation budget you have.
- Measure agreement between the labeller and humans on that set, and compare it against how often two humans agree with each other, not against 100 per cent. The ceiling derivation on reward models applies unchanged.
- Debias the mechanics you can. Randomise and average over presentation order. Control for length when constructing pairs. Do not use the policy model as its own labeller if you can afford another one.
- Use exact checks where they exist. For anything with a right answer, a verifiable reward beats any judge, model or human. Reserve preference labelling for the genuinely subjective part.
- Write the principles down and version them. If a labeller is applying a criterion, that criterion is now a configuration file with a large downstream effect. Treat it like one.
The reasonable summary is that AI feedback is a scaling technique with a known and specific hazard, rather than either a free lunch or a shortcut to avoid. It buys volume and consistency; it charges in correlated error; and the exchange rate depends entirely on how good the labeller is at the particular judgement you are asking it to make.