ARC-AGI: The Test Built to Resist Memorisation

ARC-AGI is a set of coloured-grid puzzles where each task follows a rule you have never seen, demonstrated by two or three example pairs, and you must apply it to a new input. It was designed so that scaling training data does not help — and the two mechanisms that make that design stick are a genuinely private test set and a published compute budget.

What ARC-AGI is

The Abstraction and Reasoning Corpus was introduced by François Chollet in the 2019 paper On the Measure of Intelligence. The paper’s argument is that a test of intelligence should measure skill acquisition efficiency — how well a system handles a task it has not been prepared for — rather than skill at tasks it was prepared for. Most benchmarks measure the second thing. ARC was built to measure the first.

Concretely, each task is a handful of input-output grid pairs. Grids are up to 30 by 30 cells, each cell one of ten colours. The transformation from input to output is a rule — gravity, symmetry completion, counting, flood fill, object selection, some composition of those — and you infer it from the demonstrations alone. There is no natural-language description of the rule anywhere.

The rules are drawn from what the paper calls Core Knowledge priors: objectness and cohesion, basic geometry and topology, numbers and counting, and goal-directedness. These are the things a young child has and a search engine does not, which is the point — you cannot look the answer up because the task is unique.

The shape of one task

Task (illustrative, not from the corpus). Grids as digit matrices,
0 = background.

  Demonstration 1        Demonstration 2
  input      output      input      output
  0 0 0      0 0 0       0 3 0      0 0 0
  0 5 0  ->  0 0 0       0 3 0  ->  0 0 0
  0 0 0      0 5 0       0 0 0      0 3 3

  Test input             Your answer
  0 0 7      ?           (the rule must be inferred from
  0 0 0                   the demonstrations alone — no
  0 0 0                   description is provided)

Scoring is exact match on the full output grid: correct size,
correct colour in every cell. One cell wrong scores zero.

The example above is illustrative rather than a real corpus item, and real tasks are considerably subtler. What it shows correctly is the interface: a few pairs, no instructions, exact-match grading.

Four sets, and why the private one exists

The private set is the mechanism. Every other benchmark on this cluster publishes its test items, and from the moment it does, contamination exposure only increases — the argument in benchmark contamination applies to all of them. ARC-AGI is one of the few that keeps a set back permanently and reports the number from it separately.

The cost of that design is real and worth stating: a private set cannot be inspected by outsiders. You are trusting the organisers about item quality and about the difficulty match between the private set and the public one. Every anti-contamination design trades transparency for integrity, and this is that trade in its purest form.

How it is scored

Exact match on the output grid, with two attempts permitted per test input. A task counts as solved if either attempt reproduces the target grid exactly — right dimensions, right colour in every cell. The score is the percentage of tasks solved.

Two attempts rather than one is a considered choice: it allows a system to hedge between two readings of an ambiguous demonstration without turning the metric into a coverage measure the way high-k pass@k does. It is effectively pass@2 with a hard cap.

The second half of the score is the budget. Verified leaderboard entries are reported with a cost per task alongside the accuracy, because an unbounded search over programs can solve grid puzzles given enough compute, and a benchmark about efficiency of skill acquisition that ignored compute would be measuring the wrong thing. This is one of the few places where a public leaderboard treats cost as a first-class axis rather than a footnote — see cost-aware leaderboards.

Why grids are hard for a language model specifically

Part of ARC-AGI’s difficulty is the reasoning it demands. Part of it is an interface mismatch that is worth separating out, because the two have different implications.

• The grid must be serialised. A model reads a sequence of tokens, so a 30 by 30 grid arrives as 900 numbers in some linear order with separators. Two cells that are vertically adjacent in the grid are 30 tokens apart in the sequence. Every spatial relationship the puzzle depends on has to be reconstructed from arithmetic on positions.
• The serialisation is a choice that changes results. Rows separated by newlines, digits with or without spaces, a JSON array of arrays, or a rendered image all present the same task differently, and models are not indifferent between them. Any reported ARC result should state the representation, for the same reason a multiple-choice result should state its extraction rule.
• Tokenisation fragments the digits. How a run of digits is split into tokens depends on the tokeniser and on the surrounding characters, so a column of identical colours may not have a consistent token representation. The general phenomenon is described in what a token is.
• Output is exact-match over hundreds of cells. The model must emit the entire grid correctly. A single-cell slip in cell 700 discards a fully correct rule inference, which means the metric mixes reasoning ability with long-form transcription reliability.

This matters for interpretation. A system that solves the puzzle conceptually but drops a cell is scored identically to one that had no idea, so a low score is a mixture of two failures. It also explains why approaches that generate a program to transform the grid, rather than the grid itself, do disproportionately well: emitting a short program sidesteps the transcription problem entirely.

What each version deliberately changed

ARC-AGI-1 (2019)

The original corpus. Its weakness, discovered over the following years, was that a substantial fraction of tasks yielded to brute-force program synthesis: enumerate a domain-specific language of grid operations, search for a program consistent with the demonstrations, apply it. That is a legitimate solution method and it is also not what the benchmark was trying to measure.

ARC-AGI-2 (2025)

Rebuilt to resist that. Tasks were selected for compositional depth — requiring several rules applied together rather than one rule applied cleanly — which explodes the search space that brute-force synthesis has to cover. The organisers also human-tested the tasks, keeping only ones that multiple people solved within the attempt limit, so that a low machine score cannot be dismissed as the tasks being unreasonable. And efficiency was promoted from a note to a reported number.

ARC-AGI-3

Announced as a move from static grids to interactive environments, where a system must act, observe the result and adapt within an episode rather than infer a static mapping. That shifts the test toward the territory covered by agentic benchmarks, with the same novelty-first design principle.

What a score does and does not license

A verified private-set score, reported with its compute budget, is one of the more meaningful numbers in public evaluation. It says: on tasks this system had no opportunity to memorise, under a stated budget, it solved this fraction. Very few benchmarks can make the first half of that claim honestly.

What it does not say is anything about language, knowledge, code, tools, or any task with a natural-language interface. ARC-AGI is a deliberately narrow instrument. A system could score highly on it and be useless at everything you would actually deploy a model for, and a system could be excellent at your work and score poorly here.

It also does not settle the question its name invites. The organisers are explicit that saturating the benchmark would not constitute general intelligence — it would mean this particular resistance to memorisation had been overcome, and a harder version would follow, as one did. Reading the score as a progress bar toward AGI is a misreading the authors themselves warn against, and it is the reading most launch coverage takes.