Benchmarks in Robotics: Why the Numbers Are So Hard to Compare

5 min read · updated August 3, 2026

Machine learning got a comparison culture almost free: a fixed test set, a metric, and a number anyone can reproduce in minutes. Robotics has none of those, and the reasons are structural rather than a matter of the field not having got around to it.

There is no held-out test set

In a robotics evaluation, the test set is a physical situation that has to be recreated, and recreating it exactly is not possible. Object poses differ by millimetres. Lighting changes with the time of day. The gripper pads have picked ten thousand parts since the last run and their friction has changed. The room temperature affects the motors.

Three consequences follow immediately.

The hardware is part of the benchmark. Two labs running the same nominal task on the same nominal robot model differ in calibration, wear, camera mounting, controller tuning and gripper pads. A number produced on one setup is a joint measurement of the policy and the setup, and the two cannot be separated after the fact.
Trials cost labour. Someone resets the scene between attempts. At thirty seconds a reset and a minute a trial, a hundred trials is around two and a half hours of a person’s time for a single condition — and a proper comparison has several conditions.
The initial-state distribution is a choice nobody publishes. “Objects placed randomly” can mean uniformly over the table or roughly where they were during training. The difference is enormous and it is almost never specified precisely enough to reproduce.

What twenty trials can distinguish

This is the arithmetic that should accompany every robot success rate you read, and it is elementary. A success rate over n independent trials is a binomial proportion, with standard error sqrt(p(1-p)/n). A rough 95% interval is ±1.96 standard errors.

SE = sqrt( p(1-p) / n ),   95% CI ~ p +/- 1.96*SE

p = 0.80
  n =  10   SE = 0.126   CI = 0.55 .. 1.00   (width 0.45)
  n =  20   SE = 0.089   CI = 0.63 .. 0.97   (width 0.35)
  n =  50   SE = 0.057   CI = 0.69 .. 0.91   (width 0.22)
  n = 200   SE = 0.028   CI = 0.75 .. 0.85   (width 0.11)

to distinguish 80% from 90% at conventional power you need
roughly 200 trials PER CONDITION. at ~1.5 min per trial with
resets that is about 5 hours of one person's time for one
comparison -- before anything goes wrong.

Read that against how results are usually reported. A policy scoring 16 out of 20 and another scoring 14 out of 20 is a difference of two successes, comfortably inside the noise, and yet it will be written up as one method outperforming another. The trial counts in robotics papers are frequently in the ten-to-fifty range because that is what the labour allows, which means most reported differences between methods are simply not resolvable by the experiment that was run.

This is not a robotics-specific innumeracy; it is the same problem as how many evaluation samples you need before a difference means anything, with the added cruelty that here each sample costs a minute of human labour instead of a fraction of a cent. The correct response is not to stop reporting; it is to report n and an interval, and to treat your own results with the same suspicion.

Simulated benchmarks measure the simulator

Simulation removes the labour problem entirely — thousands of trials, reproducible seeds, identical initial states — and introduces a different one. A leaderboard established in simulation ranks policies by how well they exploit that simulator’s contact solver, its friction model, its rendering and its sensor idealisations.

For tasks where the gap is narrow — free-space motion, navigation, gross dynamics — the ranking is informative. For contact-rich manipulation it can invert, because the policy that best exploits a soft-contact approximation may be the one that transfers worst. The honest use of a simulated benchmark is as a fast screen that precedes hardware evaluation, and as a regression test against your own previous versions, where the simulator being wrong in a fixed way cancels out.

The benchmarks that exist

Benchmark	Description
Meta-World	A suite of distinct simulated manipulation tasks on a common arm, designed for multi-task and meta-learning research with defined train/test task splits. Its value is the task split — it asks about generalisation to unseen tasks rather than about peak performance on one.
RLBench	A large collection of simulated manipulation tasks with demonstrations available, built to support learning from demonstration at scale. Broad task variety on a single simulated platform.
CALVIN	Aimed at long-horizon, language-conditioned manipulation: sequences of instructions rather than single tasks. Targets the compositional failure that short-horizon benchmarks miss entirely.
LIBERO	Built around lifelong and continual learning — how knowledge transfers across a stream of tasks, and how much is forgotten. Concerned with the transfer question rather than with absolute success.
BEHAVIOR-1K	Household activities defined at a semantic level in richly simulated homes, deliberately long-horizon and diverse. Ambitious in scope, and correspondingly hard to saturate or to compare cheaply.
robosuite	A modular simulation framework with standardised task environments, used as much as shared infrastructure as as a leaderboard. Shared infrastructure is arguably the more valuable contribution.
NIST task boards	A physical rather than simulated standard: standardised assembly boards with connectors, fasteners and wiring, so that different labs can run comparable hardware tasks. Directly attacks the 'the hardware is part of the benchmark' problem, at the cost of covering a narrow slice of manipulation.

What a usable report contains

If you are producing robot results, this is the minimum for somebody else to know what you measured. If you are reading them, this is the checklist of what is usually missing.

Trial count and an interval. n per condition, and the confidence interval implied by it. A bare percentage without n is not a result.
The initial-state distribution, stated precisely. The range over which object poses were sampled, and how the sampling was done. This is the single most consequential detail and the most frequently vague.
The reset protocol. Who reset, to what tolerance, and whether a human touched anything else in between.
The success criterion. Written before the trials. “Grasped and lifted” and “placed within a centimetre of the target” are different tasks with different numbers.
What counts as a run. Whether failed attempts due to hardware faults were excluded, and how many there were. Excluding them is often reasonable and always needs stating.
The hardware and its state. Robot, gripper, pads, cameras and their placement, calibration date. Wear is a variable.
Who ran it. Whether the same person collected the training data and ran the evaluation, and whether the operator knew which condition was which. Blinding is rare in robotics and the experimenter is very much in the loop.

The related concern about training on the evaluation set applies here too and is easy to miss: contamination in robotics looks like tuning the policy on the same object instances, poses and lighting used for the reported evaluation, which happens almost by default unless somebody deliberately holds a set back.