How to Read an AI Paper

Most people read a paper the way they read an article: start at the top, continue until confused, stop. That fails on papers because a paper is not written to be read linearly — it is written so that a reviewer can find the claim, the evidence and the caveat in three different places. Read it in the order it was assembled.

The three-pass method

The three-pass approach comes from S. Keshav’s short note How to Read a Paper, written for computer science generally, and it survives contact with machine learning almost unchanged. The idea is that you read the same paper up to three times, each pass with a different question, and you are allowed — encouraged — to stop after any of them.

• Pass one asks: what is this, and do I care? Five to ten minutes.
• Pass two asks: what did they do, and does the evidence support it? Up to an hour.
• Pass three asks: could I rebuild this, and what would break? Several hours, and almost never necessary.

The discipline is in the stopping. Ninety percent of papers deserve pass one only. The skill being trained is not comprehension; it is triage.

Pass one: five minutes

Read the title, the abstract, the section headings, the figure captions, and the conclusion. Do not read the body. Then answer four questions in your head:

• What kind of paper is it? A new method, a new benchmark, an analysis of existing systems, a survey, or a position paper. These are read completely differently and the abstract almost never says which.
• What is it being compared against? The baseline is chosen by the authors, and it is the single most informative choice in the paper. A strong, current, well-tuned baseline signals confidence. A baseline that is two years old, or one the authors tuned less carefully than their own method, signals that the margin may be an artefact of effort rather than of the idea.
• What is the headline number, and on what? Note both halves. A number without its evaluation set is not information.
• Is there an ablation section? Check the table of contents or scan the headings. Its presence or absence tells you a great deal before you read a word of it — see why the ablation is the section to read first.

If you cannot answer the second question after five minutes, that is itself the finding. A paper that makes its comparison hard to locate usually has a reason.

Pass two: an hour

Now read the body, but still not the proofs and not the derivations. You are checking whether the structure of the argument holds. Three things to do, in this order.

Read the tables before the prose

Figures and tables carry the actual result; the prose is the authors’ interpretation of it. Reading the tables first means you form your own reading before you are told one. Look at what is bolded, and check whether the bolded entry is actually the best in its column — it is not always, and the caption sometimes explains why in a way the abstract does not.

Check what was held fixed

Every comparison in machine learning is a comparison under a budget: the same parameter count, the same training tokens, the same wall clock, the same FLOPs, the same hardware. Find the sentence that says which. “We use the same compute budget” controls for how much arithmetic was spent and controls for nothing else — not the data mixture, not the hyperparameter search, not the number of runs discarded before the reported one. It is a real control and a narrow one, and papers frequently lean on it as though it were total.

Mark what you do not understand

Keep a list rather than stopping. Most of what is confusing on pass two is resolved by a definition three pages later or in the appendix. If the list is still long at the end, that is a signal about the paper’s clarity as much as about your background.

Pass three: the appendix

Pass three is where you would reimplement the method, and you will almost never do it. But there is a cheap version of it that is worth performing on any paper you intend to rely on: read the appendix.

The appendix is where the caveats live, and this is a structural fact about how papers are produced rather than an accusation. The main body has a page limit and a narrative; anything that complicates the narrative but is required for honesty gets pushed to the back. So the appendix is reliably where you find the hyperparameter search that was run for the new method and not for the baseline, the tasks where the method did not help, the number of random seeds, the prompt templates, the human annotation instructions, and the sentence beginning “we found that results were sensitive to”.

A useful habit: after reading the abstract, jump straight to the appendix headings. If there is an appendix section called something like “additional results” or “negative results”, read that before you read the body. It will tell you the paper’s real scope faster than the introduction will.

A worked example

Take Attention Is All You Need, the 2017 paper that introduced the transformer. Pass one: the title and abstract say the contribution is an architecture that dispenses with recurrence and convolutions entirely, evaluated on machine translation. The baselines named are the recurrent and convolutional sequence models that were then standard, and the headline numbers are BLEU scores on WMT 2014 English-to-German and English-to-French. Section headings show a model description, a section arguing for self-attention on complexity grounds, training details, and results. That is a method paper with a real comparison. Five minutes, and you already know what it claims and against what.

Pass two: the results table shows the two model sizes, base and big, against the prior systems, along with the training cost in FLOPs — the held-fixed quantity is made explicit, which is a good sign. Then comes the table of model variations, which is the ablation: number of attention heads varied, key dimension varied, dropout varied, positional encodings swapped for a learned variant. That table is where you learn that single-head attention was worse and that too many heads was also worse, which is a much more informative statement than “multi-head attention helps”. Reading it first would have told you what the paper’s claim actually rests on.

Pass three, cheaply: the training section states the hardware and the training duration, which is what makes the result reproducible in principle. Notice also what the paper does not claim. It does not claim language modelling results, it does not claim scaling behaviour, and it describes an encoder-decoder model — while the architecture that took over the field is decoder-only. The gap between what a paper says and what it is remembered for is the most common reason people misquote it. There is more on that in the section-by-section reading of that paper.

Knowing when to stop

The instinct that ruins reading is completionism. Three rules that help:

• Stop at pass one unless the paper changes a decision. If knowing the result would not alter anything you build or believe, you have already extracted its value.
• Stop when the ablation is missing and the claim is large. A big claim with no decomposition of where it comes from is not a paper you can evaluate at pass two, so pass two is wasted.
• Stop reading and start running. For anything you would actually deploy, half an hour on your own data beats three hours in the paper. The paper tells you whether the idea is worth testing; only your evaluation tells you whether it works for you — which is the argument in why public benchmarks do not predict your results.