Preprints, Peer Review and arXiv Culture

In machine learning, essentially everything appears on arXiv first, and a large share of what people read and cite has never been peer reviewed. Working out what that costs you requires knowing what review was catching in the first place, which is less than most people assume and more than nothing.

What arXiv is

arXiv is a preprint server. Authors post papers directly; the relevant categories here are cs.LG, cs.CL, cs.AI and stat.ML. There is moderation — submissions are screened for being on-topic and being recognisably scholarly, and new authors need an endorsement — but there is no evaluation of whether the work is correct, novel or well done. Nobody checked the numbers.

This is not a defect; it is the design. The point is fast, open, timestamped distribution. It works well enough that the field’s de facto publication venue is a preprint server, and conference acceptance functions as a subsequent stamp on a document people have already been reading for months.

The practical consequence is that “it is on arXiv” conveys no quality information whatsoever, and an arXiv identifier in a reference list looks exactly as authoritative as a journal citation to a reader who does not know the difference. A great deal of confident secondhand reporting rests on that visual equivalence.

What peer review actually does

Being precise here matters, because both over-trusting and dismissing review are common. Reviewers typically read a paper for a few hours, without running the code, without the data, and without attempting to reproduce anything. Within that constraint, review reliably catches a specific and narrow set of problems:

• Missing or unfair baselines. A reviewer who works in the area knows what should have been compared against, and asking for it is the most common substantive review demand.
• Claims exceeding the evidence. An abstract asserting more than the results section supports is the standard target of a revision request.
• Missing ablations. “Which component is responsible for the gain?” is a reviewer question so routine that its absence in a published paper is itself informative.
• Undisclosed setup. Hyperparameters, dataset splits, number of runs. Much of what appears in appendices is there because a reviewer asked.
• Prior work not cited. Including the case where something very similar was published earlier, which authors sometimes genuinely do not know.

And what it does not catch: fabricated data, undisclosed contamination, errors inside code nobody ran, results that will not replicate under a different seed, and — most commonly — a real result that simply does not generalise beyond the setting tested. Review is a check on the argument as presented. It is not verification.

How ML publishing works

The field publishes at conferences rather than journals, on annual cycles, with reviewing done at very large scale by a pool that includes many junior reviewers because the volume demands it. Some venues use open reviewing, where the reviews, the author responses and the decisions are published alongside the paper.

Where open reviews exist, read them. It is the single highest-value habit available to a non-specialist. The reviews tell you the objections of people who know the area, and the author response tells you how the authors handled being pushed on their weakest point. That is information you cannot get from the paper, because the paper is the version written after all of it.

On the reliability of acceptance itself: a well-known experiment run at a major machine learning conference had two independent programme committees review an overlapping portion of submissions, and the organisers reported substantial disagreement — a large share of papers accepted by one committee were rejected by the other. A later repetition found a broadly similar picture. The reasonable reading is that acceptance is a noisy signal, that rejection says less about a paper than people assume, and that neither should be treated as a verdict on correctness.

Read the version history

Every arXiv paper carries its submission history, and it is under-exploited evidence sitting in plain sight.

• A v1 with no successors, months old — nothing was revised and, quite possibly, nothing was reviewed. Not damning; worth knowing.
• A v2 or v3 landing near a conference cycle — usually the camera-ready after review. If the venue is named on the paper, the work has passed a check.
• A comment field naming a venue — authors add this themselves, and it is the fastest way to tell whether what you are reading was accepted anywhere.
• Substantial numeric changes between versions — worth looking at directly. Corrections happen, and a paper whose headline number moved between v1 and v2 is a paper whose v1 is still being cited by everyone who read it early.

That last point is the practical hazard of preprint culture. The blog posts, the threads and the secondhand summaries are all written against v1, and none of them update. If a paper matters to you, fetch the current version rather than the one your source read.

Weighting a preprint

You cannot verify most of what you read, so you need a policy rather than a verdict. A workable one, in rough order of how much weight each signal deserves:

• Is there a runnable artefact? Code and weights beat any credential. A checkable claim is in a different category from an uncheckable one.
• Does the paper report where its method fails? A negative result, a task where the gain vanished, or a stated limitation that costs the authors something is the strongest available signal of good faith.
• Are the baselines current and tuned? The most common way to be wrong without lying, as the replication literature keeps showing.
• Has anyone independent reproduced it? Worth more than publication venue, and usually findable.
• Peer review, last. A useful but weak positive signal: it means somebody in the field read it and did not object, which is worth something and is not verification.

The default posture that follows is neither trust nor dismissal. Treat a preprint as a claim with an argument attached, weight it by how checkable it is, and reserve belief for the things that have been checked — which is roughly what the reproducibility levels are for.