InstructGPT and the Birth of RLHF
4 min read · updated August 3, 2026
A pre-trained language model does not answer questions. It continues text, and a question is often best continued by more questions. The gap between that behaviour and a usable assistant is what Training language models to follow instructions with human feedback (2022) closed, and the method it describes is still the backbone of how these systems are made.
The problem: capable but useless
The paper frames the issue as a mismatch of objectives. Next-token prediction on internet text is not the objective “be helpful, honest and harmless when a user asks you something”. The model that best predicts the corpus will happily produce a plausible list of further questions, a made-up citation, or an unhelpfully literal reply, because all of those are things that follow text like yours somewhere in the training data.
The proposed fix is to stop trying to specify good behaviour and instead learn it from human comparisons. That reframing is the contribution. Everything mechanical in the paper follows from the decision to treat “what people prefer” as something to be fitted rather than written down.
The three stages
The pipeline has three steps, and confusing them is the most common error in secondhand accounts. Only the third is reinforcement learning.
| Stage | Description |
|---|---|
| 1 · supervised fine-tuning | Human labellers write demonstration answers to prompts, and the pre-trained model is fine-tuned on those demonstrations. Ordinary supervised learning. This alone gets you most of the way to a model that responds rather than continues, which is why instruction tuning without any RL is a viable method on its own. |
| 2 · reward model | For a given prompt, several model outputs are sampled and labellers rank them. A separate model is trained on those rankings to predict which output a human would prefer. The move from rating to ranking matters: humans are far more consistent at saying which of two things is better than at assigning an absolute score. |
| 3 · reinforcement learning | The fine-tuned model is optimised against the reward model using PPO, with a penalty on divergence from the supervised model. That penalty is not decoration — without it the policy drifts into text that scores well on the learned reward and is degenerate to a human, which is reward hacking in its purest observable form. |
Two of these stages are humans producing data and one is optimisation. That ratio is the honest summary of what alignment work consisted of, and it is why the labelling instructions matter as much as the algorithm — a point the paper takes seriously, describing labeller selection, the screening for agreement with researchers, and the fact that preferences were collected from a particular, small group of people.
The result worth remembering
The headline is a preference comparison, and it is startling in the right way: labellers preferred the outputs of a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3 it was derived from. Over a hundred times smaller, and preferred.
Sit with what that does and does not say. It does not say the small model knows more, reasons better, or would score higher on a knowledge benchmark — it would not. It says that on the distribution of prompts real users submitted, judged by trained labellers on helpfulness, the aligned small model won. The lesson generalises well beyond this paper: an enormous amount of what users experience as capability is actually instruction-following, formatting and calibration, none of which scale buys you automatically.
Notice also the shape of the evidence. The evaluation is human preference on a prompt distribution drawn from actual API traffic, not a static academic benchmark. That makes it more relevant and less comparable — nobody else can run the same evaluation. Both halves of that trade-off are worth recognising when you meet it elsewhere; it is the same trade discussed in when human evaluation is still required.
The alignment tax
This is the section that gets dropped from summaries, and it is the one that shows the paper is being straight with you. The RL stage caused performance regressions on several standard NLP benchmarks relative to the base model. Aligning the model made it worse at some things.
The paper names this an alignment tax, reports it, and describes a mitigation: mixing pre-training gradients back into the RL updates, a variant it labels PPO-ptx, which reduces the regressions while keeping most of the preference gains. It does not claim the tax was eliminated.
Two things follow. First, as a reader: a paper that reports where its method hurt is telling you the evaluation was broad enough to notice, which is itself evidence of care. A paper reporting only wins has either got lucky or has not looked. Second, as a practitioner: the tax is why the strongest model on a leaderboard and the most pleasant model to use are frequently not the same model, and why a benchmark regression between two versions of an assistant may be a deliberate trade rather than a mistake.
What the paper is careful about
- Aligned to whom. The paper is explicit that the model is aligned to the preferences of its labellers and to the researchers who wrote the instructions, not to any general human value. It says so rather than letting the reader assume otherwise, and that candour is the reason the sentence is quotable at all.
- Still makes things up. Instruction-following does not fix factuality. The method optimises for what raters prefer, and raters prefer confident fluent answers, which is a mechanism that pushes in the wrong direction on agreeableness and does nothing directly about fabrication.
- Reward models are proxies. The KL penalty exists because optimising a learned proxy hard enough will find its flaws. Every subsequent debate about reward hacking, and much of the interest in methods that avoid a separate reward model, starts here.
The natural sequel is the attempt to replace most of the human preference labour with model-generated feedback against a written set of principles, which is the subject of the Constitutional AI paper.