Sim-to-Real Transfer: What Crosses the Gap and What Does Not
6 min read · updated August 3, 2026
“The sim-to-real gap” is usually said as though it were one fog. It is not. It is a short list of specific physical phenomena that a simulator approximates, and knowing which of them your task depends on tells you in advance whether transfer will be easy or hopeless.
Why simulate at all
Because it is the only data source whose cost curve points the right way. A physics simulator runs faster than real time, in parallel, without breaking, without resets that need a human, and with perfect access to ground-truth state that no sensor could give you. The massively-parallel results from around 2021 — Rudin and colleagues training legged locomotion policies in minutes using thousands of environments stepped simultaneously on a single GPU — made the point sharply: the wall-clock cost of an experience step in simulation had fallen far enough to change what was worth trying.
The catch is that a simulator computes what its equations say, and the equations are a model. The gap is not a bug to be fixed; it is the residual between a model and the world, and the engineering question is which parts of the residual your policy is sensitive to.
What transfers nearly free
- Kinematics and geometry. Link lengths, joint limits, workspace shape, collision geometry. These are design data. If the URDF is right, the simulator is right, and a policy that learned where things are will find them.
- Gross rigid-body dynamics. Free-flight motion, ballistic trajectories, inertia-dominated swinging. Away from contact, the equations of motion are well understood and a good integrator reproduces them.
- Vision, if you randomise hard enough. This is the most surprising one and it is well-supported: the domain randomisation work from Tobin and colleagues in 2017 showed policies trained purely on deliberately unrealistic randomised renderings transferring to real images, because a model trained across a distribution wide enough to contain reality treats reality as one more variation. Texture, lighting, camera pose and colour are all randomisable.
- High-level structure. Task sequencing, navigation among known obstacles, motion planning in free space. If the hard part of your problem is combinatorial rather than physical, the simulator is telling the truth.
Contact and friction: the hard part
Contact is where rigid-body simulation stops being physics and starts being a numerical policy decision, and it is the single largest source of sim-to-real failure in manipulation.
The problems compound. Rigid contact is a discontinuity: the dynamics switch mode at the instant of touch, and integrators do not enjoy that. Real contacts are not rigid — surfaces deform microscopically, and the contact patch has an area that a point-contact model does not represent. Rigid-body contact problems with Coulomb friction can be formulated as complementarity problems that in general have multiple solutions or none; the classical demonstration is the Painlevé paradox, in which a rod sliding on a rough surface admits no consistent rigid-body solution. So every simulator makes a choice — soft contact with a penetration-dependent force, a relaxed complementarity solver, a pressure-field formulation like Drake’s hydroelastic contact — and each choice is wrong in a different way.
Friction itself is worse than the model. The Coulomb model has one coefficient; real surfaces have a static coefficient higher than the kinetic one (stiction), a dependence on normal pressure, a dependence on contact history, and a strong dependence on contamination — dust, skin oil from the last person who handled the part, humidity. Nobody can tell you the friction coefficient between your gripper pad and your workpiece today to better than a wide interval, and the same object picked at 9am and 5pm may not behave the same.
The practical rule: if the task’s outcome is decided during sustained contact — insertions, sliding, in-hand reorientation, anything with a deformable — expect the simulator to be qualitatively right and quantitatively unusable. If contact is brief and the task tolerates centimetres, expect transfer. This is most of why manipulation resists learning.
Actuators, latency and noise
Three more residuals, all of which are routinely forgotten and all of which are cheap to fix once named.
Actuator dynamics
A simulator will happily apply exactly the torque a policy commands, instantly. A real actuator has a current controller with its own bandwidth, a gearbox with backlash and friction, thermal derating, and in series-elastic designs a spring whose dynamics are a first-class part of the system. The classic answer is Hwangbo and colleagues’ 2019 work on the ANYmal quadruped, which trained a small neural network — an “actuator net” — on real data to map commanded position and joint history to the torque the real actuator would actually produce, then dropped that learned model into the simulator in place of the idealised one. It is a good template: when a component is hard to model analytically and easy to instrument, learn the component rather than the whole task.
Latency
Simulators are usually zero-latency by default: the observation is the state at this instant and the action applies at this instant. Real stacks have camera exposure and readout, driver buffering, bus transport, controller scheduling and actuator response, and the total is easily one to several control periods. A policy trained without delay learns a reflex it cannot execute. The fix is to model it explicitly — inject a delay of the measured magnitude into the simulated loop, and randomise it, so the policy learns something robust to the jitter as well. See where the milliseconds go for how to measure the number you should be injecting.
Sensor noise
Depth cameras do not return depth on transparent, specular or dark surfaces; they return holes and flying pixels. IMUs drift. Encoders quantise. Force-torque sensors have offsets that walk with temperature. A simulator that renders perfect depth is training a policy on a sensor nobody owns, and the failure appears exactly on the objects that matter — glass, steel, black plastic.
The four techniques, and their prices
| Technique | Description |
|---|---|
| Domain randomisation | Randomise masses, frictions, latencies, textures and camera poses across episodes so the policy must be robust to the whole range. Price: a policy robust across a wide range is conservative on any particular instance — it moves slower and grips harder than one tuned to the truth. Widening the range too far can prevent learning altogether, which is why automatic curricula that expand the range as competence grows exist. |
| System identification | Measure the real parameters — masses, inertias, friction, delays — and set them in the simulator. Price: engineering time per robot, and it decays as hardware wears. Best combined with randomisation around the identified values rather than used alone. |
| Learned component models | Replace the least trustworthy analytic component with a network fitted to real data, as in the actuator-net approach. Price: you need real data from that component, and the model is valid only over the range you excited during collection. |
| Teacher-student / privileged learning | Train a teacher in simulation with access to ground-truth state that no sensor provides — terrain height, friction, contact forces — then train a student to reproduce the teacher's behaviour from the observations a real robot actually has. Lee and colleagues used this structure in 2020 for quadrupedal locomotion over rough terrain. Price: two training stages, and the student can only ever recover what is inferable from real observations. |
None of the four is a substitute for the last step, which is running on hardware and measuring. Simulation narrows the search; it does not settle it. The pages on what simulators differ on and why robotics results are hard to compare are both about not letting a simulator’s number stand in for the world’s.