How the Transformer Paper Came About

Attention Is All You Need appeared on arXiv on 12 June 2017 and at NIPS that December. It is a machine translation paper, its central argument is about training time rather than about quality, and the architecture it describes is not the one used by any current large language model. Almost everything that made transformers matter was added by other people within the following eighteen months.

The problem in 2017 was throughput

By 2017 the state of the art in machine translation was a recurrent encoder-decoder with attention: read the source sentence one token at a time with an LSTM or GRU, and generate the target the same way, with an attention mechanism letting the decoder look back at the encoder’s states.

It worked and it was unbearably slow to train, for a structural reason rather than an implementation one. A recurrent network’s hidden state at position t depends on the state at t−1. That dependency chain cannot be parallelised along the sequence, so a training example of length 50 requires 50 sequential steps regardless of how many processors are available. On hardware whose entire advantage is doing thousands of things at once, this is the worst possible shape of computation.

Several groups were attacking it by replacing recurrence with convolution — ByteNet and ConvS2S are both cited in the paper for exactly this. Convolution parallelises, but the number of operations needed to relate two positions grows with the distance between them: linearly for ConvS2S, logarithmically for ByteNet. The transformer paper’s pitch is that self-attention relates any two positions in a constant number of operations, which is the specific claim its table of complexities is there to support.

The argument the paper actually makes

Read the abstract and the case is not that attention produces better representations in some deep sense. It is that dispensing with recurrence and convolutions entirely gives a model that is more parallelisable and requires significantly less time to train, while being at least as good.

The three-column comparison the paper's argument rests on
(per layer, for a sequence of length n and representation size d):

                        complexity      sequential   max path
                        per layer       operations   length
  self-attention        O(n² · d)       O(1)         O(1)
  recurrent             O(n · d²)       O(n)         O(n)
  convolutional         O(k · n · d²)   O(1)         O(log_k n)

The middle column is the whole argument. Self-attention needs a
constant number of sequential steps regardless of sequence length,
so the work fits on parallel hardware.

Note the first column, because it is the bill that came due later:
self-attention is quadratic in sequence length. In 2017, with n ≈ 50
tokens of a sentence and d = 512, the n² term is negligible. It is
why context length is expensive now.

That last point deserves emphasis, because it explains why a design decision that was obviously correct in 2017 became the central engineering problem of the 2020s. At sentence length, quadratic attention costs nothing. At a hundred thousand tokens it dominates everything, and an entire subfield exists to work around it.

The paper’s other components are stated matter-of-factly and have all survived: multi-head attention, so that different heads can attend to different kinds of relationship simultaneously; scaled dot-product attention, with the division by the square root of the key dimension to stop the softmax saturating; sinusoidal positional encodings, since a set of vectors with no recurrence has no inherent order; residual connections and layer normalisation around each sub-layer; and a position-wise feed-forward network with an inner dimension four times the model dimension. That 4× ratio is still the default in models a thousand times larger.

What it reported

Eight GPUs for three and a half days. The paper that produced the architecture behind every frontier model was a result anybody with a single well-equipped machine could have reproduced in a week, and this is worth stating plainly against the current cost of a frontier training run.

The paper also reports an English constituency parsing experiment, included to show the architecture generalised beyond translation. It is a modest result and it is the only evidence in the paper that the model is good for anything other than the task it was built for.

The ablation table, which is the interesting part

The paper contains a table of variations on the base model, and it is the section that best repays reading now, because it shows which of the design choices the authors themselves found load-bearing.

• Head count has an interior optimum. Keeping the total attention computation fixed and varying the number of heads, single-head attention is worse than the base configuration, and so is going to a very large number of heads. Splitting the representation into too many small subspaces costs more than the diversity of attention patterns gains.
• Reducing the key dimension hurts. The authors note that this suggests determining compatibility between positions is not easy, and that a more sophisticated compatibility function than a dot product might help. That remark is one of the paper’s few pieces of speculation and it has been the subject of a good deal of subsequent work.
• Bigger is better and dropout matters. Larger models do better, and removing dropout makes the results worse, on a dataset of this size. Both are unsurprising and both are reported rather than assumed, which is more than most architecture papers of the period managed.
• Learned positional embeddings perform about the same as the sinusoidal ones. The paper reports nearly identical results for the two, and says it chose sinusoids on the hypothesis that they might extrapolate to sequence lengths longer than those seen in training. That hypothesis turned out to be optimistic — length extrapolation remained a genuine problem, and the positional encoding schemes in use now are neither of the two the paper compared.

The general lesson from that table is that the transformer as published is a set of choices, several of them made on modest evidence at one model size on one task, and that the field then inherited all of them at a thousand times the scale. Some survived scrutiny and some — the positional encoding in particular — did not.

What is not in the paper

This is the section that separates the source from the folklore. The following are all routinely attributed to Attention Is All You Need and none of them is in it:

• The decoder-only architecture. The paper’s model is an encoder-decoder, because translation has a source and a target. Every current large language model is decoder-only. That configuration was introduced separately — a 2018 paper on generating Wikipedia articles by summarising source documents used a decoder-only transformer, and GPT-1 used one in the same year.
• Pretraining and fine-tuning. The models in the paper are trained from scratch on the translation task. The idea of pretraining a transformer on unlabelled text and then adapting it is from GPT-1 and BERT, both in 2018.
• Any claim about scale. There is no scaling argument in the paper. It reports two model sizes and does not extrapolate. The scaling law literature begins in 2020.
• Language modelling as a general interface. Nothing in the paper suggests that next-token prediction on a large corpus produces general capability. That is a 2019 and 2020 observation.
• Tokenisation as we now do it. The paper uses byte-pair encoding and word-piece vocabularies of 32,000 to 37,000 tokens for translation. The vocabularies and the handling of code, multilingual text and whitespace all changed substantially afterwards.

The accurate one-line summary is: the paper contributed the block, not the recipe. That is not a diminishment — the block turned out to be the reusable part, and section-by-section it holds up remarkably well. A closer reading of the architecture itself is in the section-by-section walkthrough of the paper.

The eighteen months that turned it into everything

1. June 2017. The transformer, for translation.
2. Early 2018. A decoder-only transformer is used for long-form generation, showing the encoder is unnecessary when there is no separate source sequence.
3. June 2018. GPT-1: a decoder-only transformer pretrained on unlabelled books, then fine-tuned per task. The architecture plus unsupervised pretraining, for the first time.
4. October 2018. BERT: an encoder-only transformer pretrained with masked language modelling, which took over every classification and understanding benchmark within months and made “pretrain then fine-tune” the default in NLP.
5. February 2019. GPT-2 removes the fine-tuning step for many tasks and demonstrates that scale plus a plain language modelling objective produces zero-shot task performance.

Five papers in twenty months, only the first of which is the one everybody cites. If you want to know where any specific property of a modern model came from, the answer is almost never 2017.

The eight authors

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. The paper carries a footnote stating that the contribution is equal and that the listing order is random — an unusual choice, and one that has complicated every attempt to describe who did what.

The title is generally reported to be a nod to the Beatles song, and the suggestion is usually attributed to Llion Jones. All eight authors had left Google by the early 2020s, and between them founded or joined a large number of the companies now working on the technology — Cohere, Character.AI, Sakana AI, Inceptive, Adept and NEAR among them, with Kaiser going to OpenAI. That diaspora is a fair proxy for how the field valued the paper in retrospect, and it is a much better-attested fact than any of the anecdotes about how the work was done, which come from interviews given years afterwards and do not always agree with one another.