The Robotics Data Problem: Why There Is No Internet of Torques

Language models were trained on a corpus that already existed and cost nothing to produce. There is no equivalent for manipulation, and the reason is not that nobody has bothered to collect it.

What is missing is the label

A policy is trained on pairs: an observation, and the action that was taken given it. The observation half is abundant — the internet has an enormous quantity of video of people doing physical things. The action half is what does not exist.

Watch a clip of someone twisting a stiff jar lid. What you can extract is a sequence of images and, with effort, an estimate of hand pose over time. What you cannot extract is the grip force, the applied torque about the lid axis, the moment the hand adjusted because it started to slip, or the internal state that decided to adjust. Those are the labels, and they are invisible in the pixels.

The formal version is that recovering actions from observations is an inverse dynamics problem, and it is under-determined for exactly the cases that matter. Two very different force profiles can produce identical visible motion — a light touch on a low-friction surface and a heavy one on a high-friction surface look the same from a camera — and the difference between them is the entire content of the skill. Video tells you the trajectory and withholds the wrench.

There is a second, quieter absence: video contains almost no failures. People film and upload the successful take. A policy learns what to do when things go right and has never seen the recovery from a dropped object, because nobody posts that.

The order-of-magnitude comparison

Put the two corpora on the same axis. Frontier language pretraining runs are described in trillions of tokens — call it 10¹² to 10¹³ units of supervision.

For robotics, the largest openly assembled dataset is the Open X-Embodiment collection released in 2023, which pooled trajectories contributed by roughly twenty institutions across more than twenty distinct robot embodiments and is described as containing on the order of a million trajectories. A trajectory is not a token; it is a whole episode. If a typical episode runs a few hundred timesteps, a million trajectories is somewhere around 10⁸ individual action steps.

language pretraining   ~1e12 - 1e13 tokens      (scraped, marginal cost ~0)
pooled robot data      ~1e6  trajectories
                       ~1e8  action steps        (produced, one at a time)

ratio                  four to five orders of magnitude
and the small one had to be paid for

Four orders of magnitude is not a gap you close by trying harder, and the marginal costs run the wrong way: the next trillion tokens of text cost a crawl, and the next million trajectories cost operator-years. This is the single fact that shapes the field, and it is why simulation is not a convenience but the only source with the right cost curve.

The four sources, and what each costs

What an hour of teleoperation buys

The costing is simple enough to do here, and worth doing because it is the number that decides project plans. All inputs below are assumptions — replace them with your own rig’s.

Assume a demonstration of a short manipulation task takes 20 seconds of motion, plus 15 seconds to reset the scene, plus occasional re-teaching, drops and hardware faults that cost another 25% overhead on average. That is roughly 44 seconds per usable demonstration, or about 80 per hour from one operator at one station.

per demo   = (motion + reset) * (1 + overhead)
           = (20 s + 15 s) * 1.25 = 43.75 s
per hour   = 3600 / 43.75 ~= 82 demos/operator-station-hour

10,000 demos for one task  = ~122 operator-hours
                           = ~3 working weeks, one station
50 tasks at that depth     = ~6,100 operator-hours
                           = ~3 operator-years, one station

Two things fall out immediately. First, parallel stations are the only lever that changes the calendar, which is why data-collection efforts look like small factories. Second, the cost is per task and per embodiment unless something transfers — which is precisely the bet robotics foundation models are making, and why that bet matters more than any individual policy result.

What the data is biased towards

• Towards success. Operators demonstrate the task working. A policy trained on it has no model of recovery, so its first mistake takes it into a state with no supervision — the compounding-error problem that behaviour cloning has to answer for.
• Towards the operator’s strategy. If the demonstrator always approaches from the left, left is what the policy learns, and the right-hand approach is out of distribution forever.
• Towards the rig. Camera placement, lighting, table height and gripper geometry are all baked in. Moving a camera two centimetres has been enough to break policies, which is a data problem masquerading as a robustness problem.
• Away from force. Most teleoperation interfaces do not record contact forces, and several — VR controllers especially — do not even give the operator force feedback, so the demonstrations contain no information about how hard anything was pressed.
• Towards short horizons. Long tasks are expensive to demonstrate and expensive to reset, so datasets skew towards episodes of tens of seconds. Everything about multi-step, long-horizon behaviour is under-supervised as a direct consequence of reset cost.

None of this is a reason not to collect teleoperation data; it is currently the best source there is. It is a reason to read any claim about a policy’s generality against the question of what its data could possibly have contained.