The GPT Papers: What Each One Introduced

Read in sequence, these documents are one argument being tested at increasing scale — and then, at the fourth step, a change in what a frontier lab is willing to publish at all. The second change matters as much as the first.

GPT-1: pre-train, then fine-tune

Improving Language Understanding by Generative Pre-Training (2018) makes a claim that is hard to remember as surprising: you can train a transformer decoder on a large amount of unlabelled text with a plain next-token objective, then fine-tune that single model on each downstream task, and beat systems built specifically for those tasks.

The part worth noticing is the machinery it needed. Because the model produced one sequence and the tasks had different shapes — entailment takes two sentences, multiple choice takes a question and several answers — the paper introduces task-specific input transformations that flatten each structure into a single token sequence with delimiters. It also still requires a labelled dataset and a separate fine-tuning run per task. Both of those are exactly what the next two papers remove, and seeing the scaffolding here is what makes the removal legible.

GPT-2: drop the fine-tuning

Language Models are Unsupervised Multitask Learners (2019) pushes the argument one step: if the pre-training corpus is large and varied enough, the model performs tasks it was never fine-tuned for, because those tasks appear naturally in the text. Ask for a summary in the prompt and you get a summary — no gradient updates, no task head.

Two details are worth carrying forward. The largest configuration was 1.5 billion parameters, and the paper reports results across the size range, which is where the habit of publishing a size sweep rather than one model comes from. And the release was staged, with the largest weights withheld initially on misuse grounds — the first prominent instance of a capability paper arriving with a release-policy argument attached. Whatever you make of that decision, it established a pattern that shapes what later papers do and do not include.

GPT-3: in-context learning

Language Models are Few-Shot Learners (2020) is the one that changed how people use these systems. At 175 billion parameters, the model performs a task from a handful of examples placed in the prompt, with no weight updates at all. The paper names the three regimes it evaluates — zero-shot, one-shot and few-shot — and reports all of them across the model sizes, so the reader can see the gap between them close as scale increases. That format is the argument: the phenomenon is a function of scale, and a table showing it at one size would not have demonstrated anything.

It is also a long paper with a substantial section on limitations and on broader impacts, including contamination analysis — checking whether evaluation data appeared in the training corpus. That analysis is genuinely hard to do well and the paper is candid about the difficulty. Contamination has since become a standing problem for every benchmark claim, which is its own subject: how contamination is detected and what it invalidates.

What GPT-3 did not do is follow instructions reliably. A raw pre-trained model continues text; it does not do what it is asked. Closing that gap is a separate paper and a separate idea — the InstructGPT line of work — and conflating the two is the most common error in casual accounts of this history.

GPT-4: the report that reports less

The GPT-4 technical report (2023) is a different kind of document, and it says so. It states explicitly that it contains no details of the architecture, model size, hardware, training compute, dataset construction or training method, citing competitive and safety considerations. That sentence is the most important one in the document for anyone trying to read the literature, because it marks the point at which the frontier stopped being externally checkable.

What it does contain is still worth reading. There is an extensive evaluation section covering academic and professional examinations, which introduced its own genre of headline; there is a discussion of predictable scaling — fitting the loss of the large run from much smaller runs — which is a real methodological contribution; and there is an accompanying system card covering safety evaluation and mitigation.

Read it as a product disclosure rather than as a paper. The distinction is not pedantry: a paper invites you to check the method, and a document that withholds the method is asking for a different kind of trust. You can still evaluate the evaluations — were the exams held out, how was contamination checked, what was the scoring procedure — and you should, because that is the only part left to check.

Reading a model card

The model card format was proposed in Model Cards for Model Reporting (2019) as a short standard document accompanying a released model: intended use, out-of-scope use, training data, evaluation data, disaggregated performance, ethical considerations, caveats. Most frontier releases now ship something in this family, often called a system card, and it is frequently the only structured information available.

A card is not a paper and rewards a different reading order:

• Read “out of scope” first. It is the only section written to constrain the reader’s enthusiasm, so it is the least marketed part of the document.
• Check whether performance is disaggregated. One aggregate number across all inputs hides the subgroups where a model is weakest, and the original proposal exists largely to make that breakdown standard.
• Note what the evaluation section does not cover. Cards list evaluations run, not evaluations considered. An absence is not evidence of a weakness, but it is the boundary of what the document supports.
• Separate claims about the model from claims about the product. Safety mitigations often live in the serving stack — filters, refusal policies, system prompts — and behaviour attributable to those does not transfer if you access the same weights elsewhere.
• Date it. Cards are revised, and a model alias may resolve to different weights over time. A card read six months ago is a card you have not read.

The trajectory across these four documents is worth naming plainly. Each step made the systems more capable and each step made them less externally verifiable, until the final one withheld the method entirely. Both halves are part of the lineage, and a reader who only tracks the first half will keep being surprised by how little there is to check.