Reinforcement Learning for Control: Sample Efficiency Is the Binding Constraint

Reinforcement learning is the natural formulation for control: there is a state, an action, a dynamics, and something you want maximised over time. It is also, in its model-free form, catastrophically expensive in exactly the currency robots cannot spend.

What RL offers that imitation does not

Three things, and each is worth real money.

• No demonstrator. The supervision comes from a reward function you write once, not from an operator producing trajectories at eighty an hour. If the reward is cheap to evaluate, the data is cheap in human terms.
• It can exceed the demonstrator. An imitation policy is bounded above by the person who taught it. An RL policy optimises the objective, and for dynamic behaviours — a gait, a throw, a recovery from a shove — the optimum is often something no human can teleoperate at all, because the human is not fast enough to demonstrate it.
• It learns recovery for free. Exploration visits off-nominal states by definition, so the policy has supervision in exactly the region where behaviour cloning has none. A locomotion policy that has been pushed ten million times in simulation knows what to do when it is pushed.

The arithmetic in robot-hours

Now the reason this is not simply the answer to everything. Model-free deep RL is usually described as needing somewhere between 10⁶ and 10⁹ environment steps for a continuous control task, depending heavily on the algorithm, the reward density and the dimensionality. Take a middling figure and convert it into hardware time.

assume  1e8 environment steps
        control rate 100 Hz  ->  100 steps per second

pure interaction time = 1e8 / 100 = 1e6 s = 11.6 days (continuous)

now add reality:
  duty cycle          40%   (resets, faults, cooldown, charging)
  reset labour        one human per station
  hardware attrition  gearboxes, cables, sensors

elapsed = 11.6 / 0.40 = ~29 days of a robot doing nothing else,
          supervised, for ONE task, with no guarantee of convergence,
          and every one of 1e8 steps executed by real actuators.

That is the whole story. It is not that RL does not work on robots; it is that a month of a supervised machine per task, plus the wear, plus the risk that the run diverges and you do it again, is not a research loop anyone can iterate on. Add that exploration means deliberately taking bad actions, and the safety story becomes its own project.

Four ways around the sample problem

Move the samples into simulation

The dominant answer, and the reason the previous page exists. Stepping thousands of environments in parallel on a GPU turns 10⁸ steps from a month of hardware into a manageable wall-clock run; the massively-parallel legged-locomotion work around 2021 demonstrated exactly this reduction. The bill is transferred rather than paid off: you now owe the sim-to-real gap, which for locomotion is largely payable and for contact-rich manipulation often is not.

Start from a policy that already works

Pretrain by imitation, then fine-tune with RL. The starting policy is already near the demonstrated behaviour, so exploration is local, the reward is encountered rather than searched for, and the number of steps needed falls by orders of magnitude. Residual RL is the sharper version of the same idea: freeze a conventional controller and learn only a small corrective term on top of it, which bounds how badly exploration can behave because the base controller is still doing most of the work.

Learn a model and plan in it

Model-based RL fits a dynamics model from interaction and then does its optimisation against the learned model rather than the robot, spending real steps only to improve the model. Sample efficiency improves markedly; the failure mode is that the optimiser finds and exploits places where the learned model is wrong, producing plans that are excellent according to the model and nonsense in the world.

Use data you already have

Offline RL trains from a fixed dataset with no new interaction, which suits robotics because logged teleoperation and deployment data accumulate anyway. Its central difficulty is distributional: the learned value function can be wildly optimistic about actions the dataset never contains, so the methods that work are the ones that constrain the policy towards the data — the same conservatism that limits how much better than the data it can get.

Reward is a specification, and specifications leak

Writing the reward is not a formality. It is writing down what you want, completely, in a form an optimiser will attack.

Sparse rewards — one at the end if the task succeeded — are honest and nearly unlearnable, because random exploration essentially never reaches the success state in a high-dimensional continuous action space. Shaped rewards add intermediate terms to guide the search and introduce the possibility that the shaped optimum is not the real one: the standard robotics examples are policies that maximise a “distance to goal” term by vibrating at high frequency, that learn to exploit a simulator’s contact solver to gain energy from nothing, or that satisfy a reaching reward by knocking the target closer. None of those are the policy misbehaving. They are the reward being an incomplete specification and the optimiser reading it exactly.

There is also a reward-availability problem specific to hardware. In simulation you can reward using privileged state — the true friction coefficient, whether the object is actually grasped, contact forces at every point. On the robot none of that is measurable, which is one motivation for teacher-student schemes where the privileged information is used only during training and distilled into a policy that runs on real observations.

Where it wins, and where it does not

The honest summary is that RL for robotics has largely become a fine-tuning and simulation technique rather than a from-scratch hardware technique, and the arithmetic above is why. The constraint is not conceptual. It is the number of times you can move a real machine.