GPT-1 to GPT-3: What Each One Added

GPT-1, GPT-2 and GPT-3 are three papers over two years, and each adds exactly one idea. Pretraining transfers. Scale removes the need for fine-tuning. Examples can go in the prompt instead of in the weights. Everything else that happened between 2018 and 2020 is those three ideas plus a hundredfold increase in parameters.

GPT-1, June 2018: pretraining transfers

The paper is Improving Language Understanding by Generative Pre-Training, by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. It was published as an OpenAI technical report rather than at a conference, which is worth noting: the most consequential model family in the field starts with a paper that was not peer reviewed.

The model is a 12-layer decoder-only transformer with about 117 million parameters, pretrained on BooksCorpus — roughly 7,000 unpublished books — with a plain next-token objective, and then fine-tuned separately on each downstream task.

The contribution is the two-stage recipe plus a specific trick that makes it practical. Rather than designing a task-specific architecture for each problem, the paper converts every task into a single sequence of tokens with delimiters — for entailment, premise then a separator then hypothesis; for multiple choice, the context concatenated with each option in turn. One architecture, one set of pretrained weights, a linear head per task. It reported improvements on nine of the twelve tasks studied.

The paper’s deeper claim, and the one that turned out to matter, is about the source of the gain: unsupervised pretraining on a large text corpus learns things that transfer, and a long contiguous corpus of books is better than shuffled sentences because the model learns to condition on long-range structure.

GPT-2, February 2019: drop the fine-tuning

Language Models are Unsupervised Multitask Learners. The largest model is 1.5 billion parameters across 48 layers — about 13 times GPT-1 — trained on WebText, a corpus scraped from outbound links from Reddit posts with at least three karma, filtered to about 8 million documents and 40 GB of text. The karma filter is the paper’s quality heuristic and it is worth remembering that the curation strategy for the corpus behind the first famous language model was “links other people found interesting enough to upvote”.

The contribution is a reframing. If a language model is trained on enough text, tasks appear in the training data implicitly — a translation appears next to its source, a summary appears after an article under the word “TL;DR” — so a sufficiently good language model performs those tasks in the course of predicting the next token. The paper tests this by evaluating zero-shot, with no fine-tuning and no gradient updates at all, and reports state of the art on seven of eight language modelling datasets in that setting.

GPT-2 is also where release policy became a subject. OpenAI initially published only smaller versions, citing concerns about misuse for generating misleading text at scale, and released the full 1.5B model in November 2019 after a staged process. The stated concern was controversial at the time and the staged release is now a normal thing for labs to do, so the episode is more significant for what it established than for what it prevented.

GPT-3, May 2020: the examples go in the prompt

Language Models are Few-Shot Learners, by Tom Brown and a long author list. 175 billion parameters, 96 layers, model dimension 12,288, 96 attention heads, a context window of 2,048 tokens, trained on roughly 300 billion tokens drawn from a filtered Common Crawl plus WebText2, two book corpora and Wikipedia.

The contribution named in the title is in-context learning. Put a few worked examples of a task in the prompt and the model performs the task, with no gradient update and no weight change. The paper distinguishes zero-shot, one-shot and few-shot settings and reports all three across eight model sizes from 125 million up to 175 billion parameters — which makes the paper as much a scaling study as a model release, and the smooth curves in it are a large part of why scaling laws became the organising idea of the following three years.

What is unusual about the paper, and what most summaries omit, is how much of it is about where the model fails. It reports weak performance on natural language inference, on some reading comprehension tasks, and on tasks requiring comparison of two sentences. It has a substantial section on training data contamination — measuring how much of each benchmark’s test set appeared in the training corpus and reporting the results, including for benchmarks where the contamination was significant. That section is the direct ancestor of the whole subsequent literature on benchmark contamination, and it was written by the people with the most to lose from it.

The three, side by side

Parameters went up by a factor of about 1,500 across two years and the architecture barely changed. GPT-3 is a 2017 transformer decoder with more layers, wider layers and alternating sparse attention patterns. Anybody who read the 2017 paper could have described GPT-3’s architecture from its parameter count. That is the finding, and it is the reason the following years were about compute and data rather than about architecture.

What stayed the same, which is almost everything

The list of things that did not change across the three papers is the finding, and it is the part a summary of the announcements will never give you.

• The objective. Predict the next token, maximise likelihood. Identical in all three. No auxiliary losses, no curriculum, no task-specific objective.
• The architecture family. A decoder-only transformer with pre-normalisation, learned positional embeddings and byte-pair encoding. GPT-2 moved the layer normalisation to the input of each sub-block and added one at the end; GPT-3 alternated dense and locally banded sparse attention patterns. Those are the changes, and they are tuning rather than redesign.
• The tokenizer. Byte-level byte-pair encoding with a vocabulary of 50,257 tokens, from GPT-2 onwards. The same vocabulary file, unchanged, across a hundredfold increase in model size.
• The absence of anything clever. No memory module, no retrieval, no explicit reasoning machinery, no symbolic component. The papers are notable for how little they add, and the authors are explicit that this is the point being tested.

Set that against the period’s expectations and it is a strong negative result about architecture research. Between 2018 and 2020 a great many proposed improvements to the transformer were published, and the model that changed the field is the plain one made larger. That observation is what the scaling law papers then formalised, and it is the empirical basis for the argument on the symbolic-versus-connectionist page.

The arithmetic that says GPT-3 was under-trained

Two years after GPT-3, a DeepMind paper on compute-optimal training — usually called Chinchilla, after the model it produced — established that for a fixed compute budget, parameters and training tokens should be scaled roughly in proportion, at something in the region of 20 tokens per parameter. GPT-3’s ratio was not close to that.

GPT-3 against the compute-optimal ratio

  parameters                 N = 175 × 10⁹
  training tokens            D = 300 × 10⁹
  ratio                      D / N = 1.7 tokens per parameter

  Chinchilla-optimal ratio   ≈ 20 tokens per parameter
  tokens GPT-3 "should" have had at that ratio:
                             20 × 175 × 10⁹ = 3.5 × 10¹² tokens

  shortfall                  3.5 trillion vs 300 billion  ≈ 12×

TRAINING COMPUTE, for reference, using the standard approximation
C ≈ 6ND floating-point operations for a dense transformer:

  GPT-3 as trained:   6 × 175e9 × 300e9   ≈ 3.1 × 10²³ FLOPs

  Chinchilla's finding is that for THAT SAME budget, a smaller model
  trained on more tokens would have been better. Solving 6ND = 3.1e23
  with D = 20N gives:
      120 N² = 3.1e23  →  N ≈ 5.1 × 10¹⁰

  i.e. roughly a 50-billion-parameter model on about 1 trillion tokens,
  for the same compute — and by the Chinchilla result, better.

ASSUMPTIONS: the 6ND approximation counts two floating-point
operations per parameter per token in the forward pass and twice that
for the backward pass, ignoring attention's quadratic term, which is
small at a 2,048-token context. The 20:1 ratio is the headline finding
of one 2022 paper and later work has argued for different ratios,
particularly once inference cost is included in the objective.

This matters for reading the history correctly. GPT-3’s headline was its parameter count, the field spent two years treating parameter count as the axis of progress, and the number turned out to be the wrong thing to have been maximising. Models released after 2022 are frequently smaller than GPT-3 and comprehensively better, because they are trained on ten to a hundred times more data. If you are reading a 2020 or 2021 article about model size, this is the correction to apply to it.

What none of the three papers is about

None of GPT-1, GPT-2 or GPT-3 is a chat model, an assistant, or a system that follows instructions reliably. The GPT-3 API launched in June 2020 and using it well meant writing prompts that a document-completion engine would continue in the direction you wanted, which is a genuinely different skill from asking for something.

The step that closed that gap is InstructGPT, published in March 2022 as Training language models to follow instructions with human feedback. It applies supervised fine-tuning on human-written demonstrations followed by reinforcement learning from human preference rankings — the recipe generally called RLHF. Its headline result is the one to remember:

This page stops at GPT-3 on purpose. GPT-3.5 and everything after it were released without papers giving parameter counts, training data or training compute, so a comparison of the same kind is not possible for them — not because the information is hard to find, but because it has not been published. That change in disclosure norms is itself one of the most significant events in the period, and any page that presents confident architecture details for post-2022 frontier models is reporting rumour.