WebArena, GAIA and Agentic Benchmarks

An agentic benchmark does not compare a string to an answer key. It puts a system into an environment, lets it act, and then inspects the environment. That is a much better measurement of whether something worked — and it makes the resulting number depend on a great deal more than the model.

What makes an agentic benchmark different

On MMLU or HumanEval the system produces one artefact and grading is a function of that artefact. In an agentic benchmark the system takes a sequence of actions, each changing the state it observes next, and grading is a function of the final state. Three things follow.

• Many correct trajectories exist. There is usually more than one route to the goal. Grading the outcome rather than the path is the only tractable choice, and it means the benchmark cannot distinguish an efficient solution from a lucky one — the trade-off is discussed in trajectory versus outcome grading.
• Errors compound. A twenty-step task where each step succeeds independently 95 per cent of the time completes 36 per cent of the time. Agentic scores are low for arithmetic reasons before any capability question arises.
• The environment is part of the benchmark. Its version, its data, its latency and its failure modes all affect the score, and unlike a static dataset an environment can drift.

Compounding, worked. Per-step success rate p, over k steps:

  p = 0.99, k = 20 -> 0.99^20 = 0.818
  p = 0.95, k = 20 -> 0.95^20 = 0.358
  p = 0.90, k = 20 -> 0.90^20 = 0.122
  p = 0.95, k = 40 -> 0.95^40 = 0.129

This assumes independence and no recovery, which is pessimistic —
a good agent detects and retries a failed step. But it explains why
a model that seems reliable turn by turn produces a low task score,
and why error recovery matters more than per-step accuracy.

The main families

The coding case has its own detailed treatment in SWE-bench and friends, including the specific problems of solution leakage and weak test suites that let an incorrect patch pass.

How an environment is scored

Reward functions in these benchmarks come in three shapes, and knowing which one applies tells you what a pass means.

State inspection is the most valuable and the most brittle. Valuable because it cannot be talked past: an agent that says it cancelled the order but did not is scored as a failure, which is exactly the failure you care about in production. Brittle because the check is written against a specific version of the environment schema, and it must be reset between tasks or one task’s side effects contaminate the next.

The score belongs to a model plus a scaffold

This is the point that distinguishes agentic benchmarks from everything else in this cluster, and it is why comparing agentic numbers across papers is usually invalid.

Between the benchmark and the model sits a scaffold: the agent loop, the system prompt, the tool definitions, the observation format, the retry policy, the stopping condition, the context management, and whatever planning or reflection steps the author added. That scaffold is often a substantial engineering artefact, and changing it changes the score on a fixed model.

See what an agent should do when a tool fails and stopping conditions for the two of those with the largest effect. The reporting consequence is blunt: a headline of the form “model X scores N on WebArena” is not a fact about model X. It is a fact about one scaffold running model X, and the scaffold may deserve most of the credit.

The reproducibility problem, itemised

• Environment drift. Self-hosted environments ship as container images, and images get rebuilt, dependencies float, and seed data changes. Two runs against nominally the same benchmark months apart may not be against the same environment. Pin the image digest.
• Network access. Whether the agent can reach the open internet during a supposedly sandboxed run changes what is solvable. Some tasks become trivial with a search engine.
• Non-determinism. Even at temperature zero, an agent over thirty steps can diverge, because the environment’s responses vary in timing and ordering. A single run is a sample. Several runs with a reported spread is a measurement — and pass^k is the right statistic for an agent that acts.
• Cost asymmetry. One system may use ten times the tokens of another for the same score. Without a cost column the comparison is incomplete — this is why ARC-AGI reports cost per task and why more benchmarks should.
• Live-web evaluation cannot be repeated at all. Not a flaw to be fixed; an inherent property. Treat those results as observations from a date, not as measurements.

Reading an agentic result

1. Find the scaffold. If the paper or post does not describe the observation format, action space and step budget, the number is not interpretable.
2. Check whether the environment version is pinned and whether the harness is published. If it is not reproducible, it is a report of an experience rather than a measurement.
3. Look for repeats and a spread. One run of a few hundred tasks with no variance estimate cannot support a comparison — and the sample-size arithmetic from GPQA applies here too, on task sets of a few hundred.
4. Look for the cost. A score without tokens or dollars is half a result.
5. Ask what the tasks resemble. WebArena is shopping and forums; GAIA is research questions; SWE-bench is Python library maintenance. If none of those is your work, the transfer argument from why benchmarks do not transfer applies with unusual force, because agentic performance is more environment-specific than any other capability measured here.