20 Papers That Explain Modern AI
5 min read · updated August 3, 2026
Reading lists usually rank by importance, which is the least useful ordering: the most important paper in a field is frequently the one you are least equipped to read. This one is ordered by dependency. Each entry says what it assumes and what it unlocks, so you can start where you are.
How this list is ordered
Four groups, roughly sequential. Foundations are the architecture and the objective. Scale is what happened when the foundations were made larger, and includes the results that redirected spending. Post-training is everything done to a pre-trained model to make it usable. Systems is how the result is served without bankrupting anyone.
The ordering is by prerequisite, not by significance, and no claim is made that reading in this order produces better comprehension than any other — it is simply that each entry becomes easier once the ones above it are familiar. If you already work with these systems daily, entering at post-training and reading backwards works fine.
For every entry: apply pass one first, and expect to stop there for most of them. A list of twenty papers is not an instruction to read twenty papers in full.
Foundations
| Paper | Description |
|---|---|
| Attention Is All You Need (2017) | The transformer. Assumes familiarity with neural networks and sequence models; unlocks everything below it. Read the model-variations table as carefully as the architecture — see the section-by-section walkthrough in this cluster. |
| Improving Language Understanding by Generative Pre-Training (2018) | Pre-train once, fine-tune per task. Assumes the transformer; unlocks the entire pre-training paradigm. Short, and useful mainly for seeing the task-specific scaffolding that later papers remove. |
| BERT (2018) | The bidirectional encoder line, trained with masked language modelling. Assumes the transformer; unlocks embeddings and classification work, and is the branch that generative modelling did not take. Worth reading precisely because it is the road not taken. |
| Language Models are Unsupervised Multitask Learners (2019) | Tasks performed without task-specific training. Assumes the previous two; unlocks prompting as a concept. |
| Language Models are Few-Shot Learners (2020) | In-context learning at 175B parameters, reported across model sizes and shot counts. Assumes the above; unlocks essentially all current practice. Long — read the setup, the size-sweep tables and the limitations section. |
Scale and its consequences
| Paper | Description |
|---|---|
| Scaling Laws for Neural Language Models (2020) | Loss as a smooth power-law function of parameters, data and compute. Assumes basic familiarity with training; unlocks the idea that capability can be forecast rather than discovered. |
| Training Compute-Optimal Large Language Models (2022) | The Chinchilla result: parameters and tokens should scale together. Assumes the previous entry; unlocks how every model since has been sized. Read the three estimation methods as an example of convergent evidence. |
| Emergent Abilities of Large Language Models (2022) | The claim that some capabilities appear abruptly with scale. Include it with its rebuttal: later work argued that the abruptness can be an artefact of discontinuous metrics, and that smoother metrics show smoother curves. Reading the pair together is a better lesson in metric choice than either alone. |
| The GPT-4 Technical Report (2023) | Read for what it withholds as much as for what it reports. Assumes nothing technical; unlocks an accurate understanding of how little of the frontier is externally checkable. |
Making models usable
| Paper | Description |
|---|---|
| Training language models to follow instructions with human feedback (2022) | The RLHF pipeline in three stages, plus the alignment tax nobody quotes. Assumes the pre-training papers; unlocks every assistant you have used. |
| Chain-of-Thought Prompting Elicits Reasoning (2022) | Intermediate steps improve multi-step accuracy, at sufficient scale, on decomposable tasks. Assumes in-context learning; unlocks the prompting literature. Read the size sweep, not just the headline. |
| Constitutional AI (2022) | Harmlessness from a written set of principles, via self-critique and model-generated preferences. Assumes RLHF; unlocks the scalable-oversight agenda. |
| Direct Preference Optimization (2023) | Optimising directly against preference pairs, without training a separate reward model or running reinforcement learning. Assumes RLHF; unlocks most open-model post-training, because it is dramatically simpler to run. |
| LoRA (2021) | Low-rank adaptation. Assumes fine-tuning basics; unlocks the entire adapter ecosystem, for the specific reason that the update merges into the weights and costs nothing at serving time. |
| Retrieval-Augmented Generation (2020) | Retriever and generator trained jointly, with the document as a latent variable. Assumes sequence-to-sequence models; unlocks a correct understanding of what the term originally meant. |
Serving and efficiency
| Paper | Description |
|---|---|
| FlashAttention (2022) | Exact attention, restructured around the memory hierarchy. Assumes the transformer; unlocks the IO-aware way of thinking about efficiency. |
| PagedAttention / vLLM (2023) | Operating-system paging applied to the KV cache. Assumes an understanding of the KV cache; unlocks modern serving throughput and prefix sharing. |
| Fast Inference from Transformers via Speculative Decoding (2023) | A draft model proposes, the target verifies, and the output distribution is provably unchanged. Assumes autoregressive generation; unlocks a large share of current latency improvements. A companion paper from another group describes the same idea independently. |
| Mixture-of-experts routing work | The line of work on sparsely activated layers, where a router selects a few experts per token. Assumes the transformer; unlocks why total and active parameter counts diverged, and why headline model sizes stopped predicting price. |
| Toy Models of Superposition (2022) | Why features are not neurons, demonstrated in models small enough to verify. Assumes some linear algebra; unlocks the interpretability literature. |
| Model Cards for Model Reporting (2019) | The documentation standard, and the shortest paper here. Assumes nothing; unlocks the ability to read a release document for what it does not say. |
Using the list
Three notes on how to spend the time, because a list this long is easy to treat as an obligation.
- Pass one on all of them beats a full read of five. Knowing what exists, what it claims and where to find it is most of the value. Depth is for the two or three that touch what you build.
- Read the pairs together. The emergence paper with its rebuttal; the RAG paper against how the term is used now; the transformer paper against the decoder-only architecture that followed. The disagreement teaches more than either half.
- Prefer these to this week’s arrivals. These have been checked, they are prerequisites for reading anything current, and the set is finite — which is the argument in following research without drowning.
One caveat about lists like this in general, and it applies to this one. A canon is a set of choices, and different practitioners would swap several entries — a computer vision engineer’s list looks very different, and so does a reinforcement learning researcher’s. This one is selected for someone building applications on hosted language models, and it is weighted accordingly. Take the dependency structure, not the membership, as the durable part.