GPQA and Google-Proof Questions

GPQA is a small set of multiple-choice questions in biology, physics and chemistry, written by people with graduate training in those fields, and filtered so that skilled non-experts with unrestricted web access still get them wrong. The filtering is the whole point, and it is also the reason the set is small enough that most reported gaps on it are inside the statistical noise.

What GPQA is

GPQA was introduced in the 2023 paper GPQA: A Graduate-Level Google-Proof Q&A Benchmark by Rein and colleagues. Around 450 questions in the main set, in three domains: biology, physics and chemistry. Each is multiple choice with one keyed answer and a handful of distractors.

The construction is what separates it from MMLU. Questions were not harvested from the web. They were written from scratch by contractors who held or were pursuing PhDs in the relevant field, and each one then went through a validation pipeline:

1. The author writes a question, the correct answer, and plausible distractors, along with an explanation.
2. Two other domain experts attempt it independently. Their answers establish whether the item is answerable by someone who knows the field.
3. Non-expert validators — people with graduate training in a different field — attempt it with unrestricted internet access and a generous time budget, typically more than half an hour per question.
4. Items are kept or discarded based on that pattern: an item that experts get right and searching non-experts get wrong is the target.

The paper reports the resulting human baselines: in-domain experts answered around two-thirds correctly, and non-experts with full web access around a third, against a chance floor set by the option count. Those are properties of the test, not of any model, and they are the numbers worth remembering — they tell you the questions are hard for people who genuinely know the field, which is unusual for a benchmark.

What “Google-proof” means operationally

It does not mean the answer is absent from the internet. It means the question was written so that finding the answer requires already knowing enough to know what to search for, and that this was checked empirically against real people with a search engine and half an hour.

That empirical check is the useful part, and it is rarer than it sounds. Most benchmarks assert difficulty; GPQA measured it against a human control group before publishing. It means a model doing well is not simply doing retrieval over material it memorised in a form a search engine would also have found.

The three subsets, and which one people quote

Diamond is the highest-confidence subset — it is the questions that most clearly satisfy the design goal. It is also under 200 items, which is where the trouble starts.

Scoring, and the sample-size problem

Scoring is plain accuracy over the items, with the same answer-extraction choices described for MMLU. There is no partial credit and no free-form grading. What makes GPQA distinctive is not the metric but the denominator.

GPQA Diamond: n = 198

One item is worth 100/198 = 0.505 percentage points.

Standard error of a proportion:  SE = sqrt( p(1-p) / n )

  at p = 0.50 -> SE = sqrt(0.2500/198) = 0.0355  -> 95% CI = ±7.0 points
  at p = 0.70 -> SE = sqrt(0.2100/198) = 0.0326  -> 95% CI = ±6.4 points
  at p = 0.85 -> SE = sqrt(0.1275/198) = 0.0254  -> 95% CI = ±5.0 points

Two systems 4 points apart on Diamond have not been distinguished.
That is 8 questions.

This is the single most useful fact about GPQA and it is almost never stated alongside a score. A four- or five-point difference on Diamond is eight or ten questions. Rerun the same model with a different few-shot seed and you can move it that far without changing anything about the model.

The interval above is the sampling interval only — it answers “if these 198 questions were drawn from a larger population of similar questions, how much would the score wobble?” It does not include run-to-run variance from temperature, which for a reasoning model on a 198-item set can be comparable. Report both, and see statistical significance with non-deterministic models for the tests that apply.

What a GPQA score licenses you to conclude

A high score, honestly obtained, supports a narrow and real claim: the model can select the correct option on hard, expert-written natural science questions that skilled searchers cannot solve by searching. In 2023 that was a strong signal. It remains a better signal than MMLU because the questions are not scraped, the difficulty was validated against people, and there are no harvested label errors of the kind MMLU carries — each item was written and checked by someone with domain training.

It does not license the conclusion that the model reasons like a scientist, because a multiple-choice format cannot separate reasoning from elimination. Nor does it license a ranking between two systems within a handful of points, for the reason above.

It also does not license conclusions about disciplines outside the three tested. GPQA is biology, physics and chemistry. There is nothing in it about mathematics — see competition maths for that — nothing about engineering, nothing about the humanities, and nothing about applied laboratory judgement.

What it is a bad proxy for

• Research capability. Answering a well-posed question with four options is not formulating a question, designing an experiment, or noticing that the premise is wrong. There is a broader treatment in benchmarks for scientific reasoning.
• Factual reliability in production. GPQA questions have a verified answer. Your users ask questions that may not.
• Anything at scale. 198 items cannot resolve small differences, and a suite you use for release gating needs to. Build your own set instead — fifty examples from your own traffic will discriminate between candidates for your task better than Diamond will.
• Cost or latency. Nothing in the score reflects what the answer took. That is a separate axis and there are leaderboards that measure it.

The right way to read GPQA is as a difficulty-calibrated sample rather than a ranking instrument. It tells you roughly where a model sits in a wide band. It cannot tell you which of two neighbours is better, and published comparisons that use it that way are over-reading their own data.