A Bullshit Detector for AI Claims

This is the summary page of the cluster: a procedure you can run in two minutes on any announcement, paper summary, thread or headline. The questions are ordered by cost, because the cheap ones eliminate most claims and the expensive ones are only worth spending on what survives.

How to use it

Run the first four on everything. If a claim survives them and you have a decision riding on it, run the next four. Run the last two only when the decision is expensive. The point is not to reach a verdict on every claim — most claims do not need one — but to know quickly which ones are worth attention.

Two rules that make the whole thing work. First, a question the source cannot answer is not proof the claim is false; it tells you the claim is unverified, which is a different and more useful status than “wrong”. Second, apply it to claims you like. A checklist used only on claims that annoy you is a device for confirming what you already think.

The four cheap questions

1 · What exactly is being claimed?

Restate it in one sentence with no adjectives. Most impressive headlines shrink dramatically under this. “Model achieves human-level performance” usually becomes “model scored within some margin of a small human sample on one benchmark under conditions the authors chose”, which is a real result and a different one.

2 · Is this a capability claim, a deployment claim or a forecast?

These need different evidence and are routinely mixed inside one sentence. “Can do X”, “is doing X in production” and “will do X by 2030” are three claims with three evidentiary standards, and the programmer-replacement claim is the standard worked example of the confusion.

3 · Who is the source and what does the primary source say?

Company blog, press release, preprint, peer-reviewed paper and news summary are different objects. The most common failure is not fabrication but drift: each retelling drops a qualifier, and by the fourth the hedged finding has become a flat assertion. Clicking through one level usually resolves it.

4 · Compared to what?

A number without a baseline is not a result. Better than the previous model, than a human, than random, than the obvious simple approach — these are wildly different claims, and the last is the one most often omitted, because a simple baseline frequently does embarrassingly well.

Four questions about the evidence

5 · How was it measured, and could the answer have been in the training data?

For any benchmark result, the conditions decide whether the number means anything, and the contamination question applies to every public test set old enough to have been crawled. Sources that address it directly are making a much stronger claim than sources that do not — why contamination is the default assumption.

6 · How big is the effect, relative to the noise?

Convert percentages back into counts. A gap of a point or two on a few-hundred-item benchmark is frequently within sampling error, and the arithmetic for how many items a difference needs takes a minute.

7 · What is the success rate, and out of how many attempts?

For anything demonstrated rather than measured, this is the whole question. One recorded run tells you a thing is possible and nothing about how often — see the gaps between a demo and a product.

8 · What is missing that you would expect to see?

The benchmark that appeared last time and not this time. The obvious comparator absent from the axis. Cost, next to a quality claim. Absence is the most informative thing in most announcements and the hardest to notice, because nothing draws attention to it.

Two questions that take real work

9 · What would have to be true for this to be wrong?

If you cannot name an observation that would falsify the claim, it is not an empirical claim — and this applies with equal force to optimistic and pessimistic framings. A prediction with no date and no disconfirming observation is a mood.

10 · Does it survive on my own inputs?

The only question that actually settles anything. Fifty examples from your own work, scored the way you care about, outweigh every chart in every launch post — because the benchmark was chosen by somebody who does not know what you are building. Building a small set of your own is a couple of hours and it is the only step here that produces evidence rather than filtering it.

Using the checklist against yourself

A checklist like this has a predictable failure mode: it becomes a tool for dismissing things, applied asymmetrically, and its user ends up more confident and less accurate. Three guards against that.

• Unverified is not false. The output of these questions is usually “I do not know yet”, and that is the correct output. Converting it to “debunked” is the same error as believing the headline, with better posture.
• Scepticism is a prediction too. “This will not work” can be scored, and in this field it has been wrong about several capabilities that arrived. Track your own calls.
• Run it on the claims you want to be true. The checklist is only worth anything if the direction of the claim does not change how hard you press.