BIG-bench, BBH and the Hard Subsets

BIG-bench is a collaborative benchmark of a couple of hundred tasks contributed by hundreds of authors. BIG-bench Hard is the subset where models of the day failed to beat the average human rater. That selection rule is the interesting part, because a task can end up in a hard subset for at least three reasons and only one of them is difficulty.

What BIG-bench is

BIG-bench — Beyond the Imitation Game — was published in 2022 by a very large collaboration: over two hundred tasks contributed by hundreds of authors across many institutions. The design goal was coverage of things existing benchmarks missed, achieved by opening submissions rather than by one team designing a test.

The tasks come in two mechanical forms.

• JSON tasks. A list of input-target pairs, scored either as multiple choice (rank the provided options) or as generation with exact or fuzzy match against targets. Most tasks are these.
• Programmatic tasks. Python that can interact with the model across multiple turns, so it can implement adaptive tests, games, or anything where the next input depends on the previous answer. A small minority, but they can express things a static list cannot.

The task list is genuinely eclectic — logical deduction, formal languages, chess move legality, joke explanation, emoji interpretation, ethical judgement, dialect translation. That is a strength for coverage and a weakness for interpretation: an average over two hundred heterogeneous tasks is even harder to read than MTEB’s average, and BIG-bench was never really meant to be consumed as a single number.

The canary string

BIG-bench includes something more benchmarks should copy: a canary string. A specific unique identifier is embedded in the benchmark files, serving two purposes.

1. Corpus builders can filter documents containing the string, so the benchmark can be excluded from training data by anyone who wants to exclude it.
2. Anyone can test a model for exposure by asking it to reproduce the canary. A model that can emit it has, at some point, been trained on a document containing it.

Neither is airtight. Filtering only works if the corpus builder chooses to filter, and a model can be trained on a copy of a BIG-bench task that was reproduced without the canary — a blog post, a paper, a GitHub issue. But it is a cheap, checkable, well-designed mechanism and it produces a positive result you can act on. The wider set of detection techniques is in benchmark contamination.

How BBH was selected

BIG-bench Hard was published later in 2022 by Suzgun and colleagues. The selection procedure was explicit:

1. Start from BIG-bench’s tasks and keep only those with an objective, programmatically checkable metric.
2. Require enough examples per task to produce a usable score.
3. Keep only tasks where no model evaluated at the time exceeded the average human-rater score reported for that task.

The result was twenty-three tasks, a few thousand examples in total. Small, cheap to run, and for a while a genuinely discriminating test — which is why it was adopted so widely and why it appears on most model cards.

What filtering on failure selects for

Here is the part worth sitting with. “Tasks all models failed” is not the same population as “hard tasks”. At least three things put a task in that bucket.

• Genuine difficulty. The task requires multi-step reasoning, careful tracking of state, or a kind of inference the models of the day could not do. This is the intended category and much of BBH is in it — multi-step logical deduction and object tracking are real reasoning tasks.
• Format hostility. The task is not hard to think about but is hard to answer in the expected shape: an unusual output format, an ambiguous instruction, a required convention the prompt does not state. Models fail on the packaging.
• Item quality. The task has mis-keyed answers, genuinely ambiguous items, or a human baseline measured under different conditions from the model evaluation. A task nobody can beat may be a task nobody can beat correctly.

A filter on failure cannot distinguish the three. It selects for all of them, in proportions nobody knows, and the second and third categories contribute noise rather than signal. This is a general property of adversarially filtered benchmarks — TruthfulQA was built by a similar filter and carries a similar critique.

The prompting method is part of the definition

BBH’s most cited finding is that chain-of-thought prompting flipped a number of its tasks from below the human baseline to above it, without changing the model. Same weights, same items, different prompt, different side of the line.

That is a genuine and useful result about prompting — it is part of why chain-of-thought prompting became standard practice. It is also a demonstration that “hard for models” was never a property of the tasks alone. It was a property of tasks under a particular prompting regime, and the regime changed within months of publication.

The practical consequence is that a BBH number without its prompting configuration is close to meaningless. Direct-answer BBH and chain-of-thought BBH are different benchmarks that share a name. Modern reasoning models, which produce extended working by default, are effectively always in the second regime whether or not the harness asks for it — so their BBH numbers are not comparable with the direct-answer figures from 2022 papers.

Reading a BBH number in 2026

BBH was selected against the capabilities of 2022 models. Those capabilities moved. A benchmark defined as “what the current generation cannot do” has a built-in expiry date, and BBH reached it: much of it is now near the ceiling, where the noise arithmetic from MMLU applies — a few thousand items spread over twenty-three tasks means a couple of hundred items per task, so per-task differences of several points are inside sampling error.

The successor pattern is exactly what you would expect: harder variants built by taking each BBH task and constructing a more demanding version of the same skill, which is what BIG-Bench Extra Hard does. Expect the cycle to continue, and expect each iteration to inherit the same selection problem.

So, when you meet a BBH figure:

• Ask whether it is direct-answer or chain-of-thought. Without that, do not compare it to anything.
• Ask for the per-task breakdown. The aggregate hides that a model may be at the ceiling on fifteen tasks and at chance on three.
• Treat the three or four genuinely hard reasoning tasks — multi-step deduction, object and state tracking — as the informative part and the rest as historical.
• Do not use it as a release gate. A few thousand items across twenty-three unrelated tasks cannot detect the regressions you actually care about; a purpose-built regression suite can.