Vision-Language-Action Models: How a Transformer Ends Up Producing Torque

A vision-language-action model is a vision-language model whose output space has been replaced. Everything interesting is in that replacement: what an action is represented as, how many of them come out at once, and what sits between the model and the motor.

The shape of the thing

Start from a familiar object. A vision-language model takes images and text and produces text; internally it is a transformer over a sequence of tokens, some of which came from a vision encoder. See how vision models fold pixels into that sequence if that part is unfamiliar.

A VLA keeps that stack and changes the head. The inputs become one or more camera views, a natural-language instruction, and usually the robot’s proprioceptive state — joint positions, gripper width, sometimes end-effector pose. The output becomes an action: for a typical single-arm setup, a seven-dimensional vector of end-effector deltas plus a gripper command, or a vector of joint position targets with one entry per joint.

Note what that output is not. It is almost never a torque. The model produces a setpoint — a position or a velocity — and a conventional controller underneath turns that setpoint into current in a motor at a much higher rate. The chain is: model to setpoint at a few hertz, controller to torque at hundreds or thousands of hertz, motor driver to current at higher still. Every layer down is faster, dumber and more deterministic than the one above, and that stratification is the whole design.

Route one: actions as tokens

The first way to make a language model emit actions is to make actions look like language. Discretise each action dimension into bins — 256 is the number the RT-2 work used — and assign each bin to a token the model already has, overwriting tokens that are rare in the text vocabulary. An action is then a short string of token ids, and generating one is ordinary autoregressive decoding.

This is a genuinely clever trick, because it means the action head costs nothing to add and the model can be co-trained on ordinary web-scale vision-language data at the same time as robot data. RT-2, in Google’s 2023 work, is the canonical example: a large vision-language model fine-tuned jointly on robot trajectories and web data, with actions expressed in the text vocabulary. The reported motivation was transfer — semantic knowledge from the web appearing in how the policy handles objects and instructions it never saw in the robot data.

The costs are equally concrete. Discretisation caps precision at the bin width: 256 bins over a ±10 cm range of end-effector delta is about 0.8 mm of quantisation, which is fine for a pick and marginal for an insertion. And decoding is sequential — one forward pass per token, so eight tokens per action is eight passes before you have one command, which pushes directly against the rate problem below.

Route two: a continuous action head

The second approach keeps the vision-language backbone as a feature extractor and attaches a separate module that outputs continuous actions directly. Diffusion Policy, from Chi and colleagues in 2023, is the widely-copied instance: rather than regressing a single action, it runs a denoising process over a sequence of future actions conditioned on the observation. Later systems use flow matching for the same job at fewer sampling steps — the π₀ architecture from Physical Intelligence is described in its technical report as a vision-language backbone with a flow-matching action expert attached, producing action chunks at 50 Hz.

Why a generative head rather than a regression? Because demonstration data is multimodal in the statistical sense. If two demonstrators went around an obstacle on opposite sides, the mean of their actions goes straight into it. A model trained with mean-squared error learns that mean. A diffusion or flow head samples from the distribution instead, so it commits to one side. This is the same failure that makes averaged predictions bad in any multimodal problem, and it is unusually easy to see in robotics because the average of two safe paths is often a collision.

The trade is that you have given up the shared vocabulary. The action head is a new module with its own parameters, trained on robot data only, and it is specific to an embodiment’s action space in a way the token route partly avoided.

Why the output is a chunk

Do the arithmetic that decides this. Suppose the full stack — image preprocessing, backbone forward pass, action head sampling — takes 150 ms. That is a 6.7 Hz policy. If the controller underneath wants a new setpoint every 20 ms (50 Hz), the policy is producing one command for every seven and a half the controller needs, and for the other six and a half the robot is executing something stale or nothing at all.

Action chunking is the answer, and it is the single most important implementation detail in modern VLAs. Instead of predicting the next action, the model predicts the next H actions — a horizon of, say, 16 to 50 steps — and the controller plays them out in sequence while the next inference runs. At 50 Hz, a chunk of 25 actions is half a second of motion, which comfortably covers a 150 ms inference. The technique was popularised as ACT, Action Chunking with Transformers, from the 2023 ALOHA work.

inference time      150 ms
controller period    20 ms  (50 Hz)
cycles per inference  150 / 20 = 7.5   <- must be covered
chunk horizon H      25 actions = 500 ms of motion
open-loop fraction   150 / 500 = 30% of the chunk is
                     consumed before the next one lands

Chunking buys the rate and spends closed-loop responsiveness: while a chunk is executing, the robot is running open-loop with respect to the model. If the world moves during those 500 ms, nothing notices until the next chunk. ACT’s temporal ensembling mitigates this by overlapping chunks and averaging the overlapping predictions, so the executed action is a blend of several recent inferences rather than a hard switch every horizon.

What it inherits and what it does not

• It inherits semantics. The backbone knows what a mug is, that “the red one” is a colour predicate, and that drawers open. This is the actual reason for building on a VLM rather than training a policy from scratch, and it is what makes instruction-following in novel scenes plausible at all.
• It does not inherit physics. Nothing in a web corpus teaches how much force a lid needs or that a wet glass slips. Those live in the robot data, which is the small part.
• It usually cannot feel. Most VLAs take vision and proprioception and no force or tactile input, which means the last centimetre before contact — the part manipulation actually turns on — is being done open-loop with respect to the only sense that matters there.
• Its action space is embodiment-shaped. A policy trained on a seven-DoF arm with a parallel gripper has a head whose output means something specific to that arm. Making one policy serve many bodies is the cross-embodiment problem, and it is unsolved in the general case.

The short version: a VLA is a semantic prior with a fast, dumb, reliable layer bolted underneath it, emitting half-seconds of motion at a time because it cannot emit milliseconds of motion at a time. Every design choice in the list above is downstream of that.