Imitation Learning and Teleoperation: Copying a Human at a Cost That Scales

Show the robot how, then have it copy you. It is the most natural idea in robot learning and it has one deep problem, which was characterised precisely in 2011 and which every modern method is still working around.

Behaviour cloning is just supervised learning

Collect demonstrations as sequences of observation-action pairs. Train a network to predict the action given the observation. Deploy it. That is behaviour cloning in full, and its appeal is that it needs no reward function, no exploration, no simulator and no safety story beyond the demonstrator’s judgement — all the things that make reinforcement learning expensive on hardware.

It also uses the same machinery as any other supervised problem, which means the whole toolbox transfers: augmentation, regularisation, held-out validation. The thing that does not transfer is the independence assumption underneath supervised learning, and that is the entire difficulty.

Why the errors compound

Supervised learning assumes the test inputs are drawn from the same distribution as the training inputs. In a control problem this is false by construction, because the policy’s own actions determine what it sees next. Make a small error, and you are now in a state slightly off the demonstrated distribution. Your prediction there is slightly worse, so the next error is larger, and so on.

Ross and Bagnell made this quantitative. For a policy with per-step error rate ε on the demonstrator’s distribution, naive behaviour cloning over a horizon of T steps has a total cost bound that grows like T²ε — quadratically in the horizon, not linearly — because the distribution shift itself grows with the number of steps taken off-distribution. Their 2011 DAgger paper (Dataset Aggregation) gives a training procedure that recovers a linear Tε bound.

per-step error  e = 0.01   (1% of steps are wrong)
horizon         T = 500    (10 s at 50 Hz)

naive cloning   ~ T^2 * e = 500 * 500 * 0.01 = 2500  (units of cost)
DAgger-style    ~ T   * e = 500 * 0.01       =    5

the ratio is T -- and T is the length of your task

The bounds are worst-case and the units are abstract, so do not read those as predicted failure rates. Read the ratio: the penalty for training only on the expert’s own states scales with the task horizon. This is the formal statement of something practitioners describe informally all the time — that cloned policies look excellent for two seconds and then wander off — and it is why long-horizon tasks are disproportionately hard for imitation.

The fix, and why robots make it awkward

DAgger’s procedure is: run the current policy, record the states it actually visits, ask the expert what they would have done in each of those states, add those labels to the dataset, retrain, repeat. The policy is thereby trained on its own state distribution rather than the expert’s, which is exactly the mismatch that caused the problem.

On a robot this is harder than it sounds for two reasons. First, the expert must label states the policy reached, which for a physical system means either annotating after the fact from logs — hard, since the expert has to imagine themselves into a configuration they did not feel — or intervening live, which changes what happens next. Second, letting a half-trained policy run to visit its own states is precisely the situation where it does something you did not want, and the cost of a physical mistake is not a wasted forward pass.

The practical descendant is intervention-based collection: a human shadows the policy on the real task and takes over the moment it starts to go wrong. Every intervention is a labelled correction in exactly the state distribution the policy generates, which is what DAgger asked for, and it doubles as the operating model for supervised deployment. It is one of the few places in robotics where the safety mechanism and the data mechanism are the same mechanism.

What each rig leaves out

The interface used to demonstrate is not neutral. Each style records a different subset of what the human knew.

The pattern is that every rig trades collection rate against label fidelity, and the thing that goes missing first is almost always force. If you later wonder why a policy presses too hard, check whether anything in the pipeline ever recorded how hard the human pressed.

Action chunking as the practical answer

The most effective engineering response to compounding error turned out to be shortening the number of decisions rather than improving each one. If the policy predicts a sequence of H future actions and executes them, a ten-second task at 50 Hz stops being 500 sequential decisions and becomes 500/H of them. With H = 25, that is twenty decisions instead of five hundred, and the quadratic term above is being applied to a horizon twenty-five times shorter.

That is the same mechanism described in how a VLA gets its output rate up, arrived at from the other direction — which is a good sign that it is the right abstraction rather than a trick. It also explains why chunk horizon is a genuine tuning knob with a real trade: longer chunks mean fewer compounding steps and more time spent blind to a world that may have moved.