Manipulation: Why Hands Are Still Hard

Getting a gripper to within a centimetre of an object is close to a solved problem. Everything that happens after that is where manipulation lives, and almost none of it is visible to the sensors that got you there.

The problem is the last centimetre

Decompose a pick. Perceive the scene, choose a grasp, plan a collision-free path to a pre-grasp pose, move there, approach, close, lift. The first four steps are geometry and search — well-conditioned, well-studied, and reliably solved by motion planners that predate learning entirely. The last three are physics involving an object whose mass, friction, compliance and internal structure you do not know.

This is why manipulation and navigation have diverged so sharply as fields. Navigation is mostly a geometric problem with a good sensor suite and a well-posed objective, and it has been productised. A manipulation error, by contrast, is not an error in where the arm is; it is an error in what happened when the arm touched something.

Contact is a discontinuity

Before touch, the arm is in free space and its dynamics are smooth, well-modelled and easy to control. At touch, a constraint appears instantaneously: forces that did not exist a millisecond ago now do, the reachable directions of motion collapse, and the system’s equations change form. Break contact and it changes back.

Mathematically this is a hybrid system — continuous dynamics with discrete mode switches — and hybrid systems are notoriously harder to control, to simulate and to learn than smooth ones. A controller tuned for free-space motion will be wrong the instant it touches something: the classic result is that a stiff position controller commanding a trajectory that intersects a surface will drive force towards infinity until something yields, which is why manipulators that touch things run impedance or admittance control rather than pure position control.

It also means the number of distinct contact configurations grows combinatorially with the number of contact points. Two fingers and a table already gives you a set of possible contact modes — sticking, sliding, separating, at each point — and the correct action differs by mode. Planners that reason about this explicitly are doing contact-mode search, and the search space is the reason it is expensive.

Friction is not a number you know

Grasp analysis has a clean classical theory. Given contact points and a friction coefficient, you can compute the friction cone at each contact, ask whether the contacts can resist arbitrary external wrenches — force closure — and rank candidate grasps by a quality metric. This theory is correct and it depends on a parameter nobody can supply.

The friction coefficient between a specific gripper pad and a specific object surface depends on the materials, the normal force, the contamination, the humidity, the temperature, the surface finish and how many parts have been picked since the pads were last cleaned. It is not one number, it is an interval, and the static value exceeds the kinetic one so the transition from stuck to sliding is itself a discontinuity. As the sim-to-real page describes, this is also why simulated contact is qualitatively useful and quantitatively untrustworthy.

The design response is to stop needing the number. Wide, soft, high-friction pads; suction, which replaces friction with pressure differential and is why so much real picking uses it; caging grasps that constrain the object geometrically rather than frictionally; and gripping hard enough that a wide error interval still holds — at the cost of crushing anything fragile.

Why vision goes blind exactly then

There is a cruel structural fact here. As the gripper closes on the object, the gripper occludes the object from the very cameras that were tracking it. A wrist camera sees mostly fingers at the moment of contact; an external camera sees mostly the back of the hand. The decisive fifty milliseconds are the ones with the worst observation.

Depth sensing degrades in the same region for a different reason. Structured-light and time-of-flight depth cameras return nothing usable on transparent, specular and very dark surfaces — the exact glass, polished-metal and black-plastic objects that a general-purpose picker must handle — so the geometry near contact is often missing rather than merely occluded.

What would fill the gap is touch. Optical tactile sensors of the GelSight family — an elastomer pad imaged from behind by a camera, so that deformation of the pad becomes an image — turn contact geometry and slip into a high-resolution visual signal, and are the most practical tactile approach in wide research use. The obstacle is not the concept; it is that tactile skins wear out, are hard to route wiring for, and, most importantly, almost no demonstration data records force at all, so there is little to train a tactile policy on even where the sensor exists.

What actually helps

• Compliance in hardware. The oldest trick in the field and still among the best: a remote-centre-of-compliance device is a passive mechanical element that lets a peg self-align into a hole under insertion force. It converts a sensing-and-control problem into a mechanics problem and it needs no computation whatsoever.
• Force control instead of position control. Commanding a force and letting position follow is the right frame once you are in contact. It also makes the safety story tractable, because a force limit is something a certified layer can enforce regardless of what the policy asked for.
• Search strategies. Spiral searches, wiggles and insertion strategies that deliberately exploit contact to localise — these are how humans insert a key in the dark, and they work for the same reason: contact is itself a measurement.
• Learned grasp quality from physical trials. The Dex-Net line of work is the well-known example of the alternative to analytic grasping: sample candidate grasps over object geometry, label them with a physics-based or empirical quality estimate, and train a network to predict grasp robustness directly from depth images. It works because it learns the distribution over uncertain parameters rather than pretending to know them.
• Reducing the variety. Fixtures, known part presentation and single-SKU flows are not defeat; they are the reason industrial manipulation works at all. Generality is the expensive axis, and it should be bought deliberately rather than assumed.

Manipulation is not waiting for a bigger model. It is waiting for a sensor that survives ten million contacts, data that records what the hand felt, and a contact model that is quantitative rather than plausible.