Literature Review With AI: A Workflow With Verification Built In

Most bad experiences with AI-assisted literature review come from treating it as one task. It is three, they fail in unrelated ways, and exactly one of them is safe to hand over almost entirely.

Three tasks, not one

Search is finding the candidate literature. The metric that matters is recall — what fraction of the relevant work you found — and the thing you cannot observe is what you missed. Screening is deciding which candidates are relevant. The metric is agreement with a careful human, and errors are visible because you can sample and check. Extraction is reading the included papers and pulling out what they actually claim. The failure here is subtle and dangerous: a plausible paraphrase of something the paper did not say.

Run all three through one chat window and you get the worst property of each: unauditable recall, unchecked inclusion decisions, and claims with no traceable origin. Split them and each becomes manageable.

It also matters enormously which kind of review you are doing, because the tolerance for a recall failure is not the same. A scoping read to orient yourself in an unfamiliar area can miss a great deal and still do its job — you are building a map, and you will correct it as you go. A systematic review cannot: its entire claim is that it found everything meeting stated criteria, so a search that silently omits a class of studies does not produce a slightly weaker review, it produces a conclusion that may be the opposite of the true one. Decide which you are doing before you decide what to delegate, because the same tool is appropriate for one and disqualifying for the other.

Search: do not delegate recall

A language model does not have an index of the literature. If it is answering from its weights, then citations are generated text and are subject to the same process that produces plausible fake references — real-sounding authors, a real journal, a title that reads exactly like a paper in that field, and no such paper. Retrieval-backed tools are better because the references exist, but they introduce a different problem: semantic search returns what is similar to your query, and you have no way to characterise what it did not return.

For anything that has to be defensible, the search itself belongs in a bibliographic database with a written query — a boolean expression, a set of databases, a date range, and the number of records retrieved. That is reproducible: somebody can run it again and get the same set. Reporting standards for systematic reviews require exactly this, and the requirement predates AI by decades because the underlying problem is not new.

Where models help at this stage is in building the query, not replacing it: generating synonym and spelling variants, suggesting subject headings, and finding a few seed papers whose reference lists and citing articles you then follow by hand. Treat any tool-found paper as a lead to confirm in the database, never as a result.

Screening: the part that works

Title-and-abstract screening is repetitive, well-specified, and the single most tedious part of a review. It is also the one stage where delegation is defensible, because you can measure the delegation.

How to set it up

• Write the inclusion criteria first, as rules. If you cannot state them precisely enough for a model, they are not precise enough for a second human reviewer either, which is a useful thing to discover early.
• Bias the model towards inclusion. A false include costs one abstract of your time. A false exclude silently removes a paper from the review and nobody ever finds out.
• Screen a labelled sample yourself first. A couple of hundred records you have decided on gives you an agreement rate, and an agreement rate is something you can report.
• Audit the exclusions, not just the inclusions. Sample the rejected pile and read it. This is the only place the expensive error is visible.
• Ask for a reason with each decision. Not because the stated reason is a faithful account of the computation — it is not — but because reading a hundred reasons surfaces a systematically misread criterion faster than reading a hundred verdicts.

Extraction: quotes, not summaries

The rule that removes most of the risk is small: nothing enters your notes as a claim about a paper unless it arrives with a verbatim quote and a location. Ask for the sentence, the section, and the page or figure. A summary with no quote behind it is a sentence the model wrote, and once it is in your notes it looks identical to a sentence the paper wrote.

This also converts an invisible failure into a mechanical one. Checking “does this summary fairly represent the paper” is an expert judgement. Checking “does this string appear in this PDF” is a search, and you can do it in seconds for every extracted claim.

Two specific things to extract deliberately, because models under-report both: the sample size and population, and whatever the paper says about its own limitations. A model asked to summarise findings will summarise findings, and the limitations section is usually where the reason not to cite it lives.

The checks, in order

What to record

Whatever you used the model for, write down which model, which version, on what date, and what you asked it. This costs a paragraph and it is the difference between a method somebody can assess and an unreproducible one — a point developed in the reproducibility of AI-assisted work. Most journals and funders now also require disclosure of generative AI use, so you will need the paragraph regardless; writing it while you work is much easier than reconstructing it afterwards.