Diffusion Language Models

An autoregressive model produces a 512-token answer in 512 sequential forward passes. A diffusion language model starts from a fully masked sequence of 512 positions and refines all of them together over a fixed number of steps — sixteen, thirty-two, sixty-four. The trade is arithmetic for sequential depth, and it is worth doing the arithmetic before believing the speed claim.

The idea: denoise the whole sequence

Image diffusion starts from noise and removes a little of it at each of many steps until an image appears. Text is discrete, so “a little noise” has to mean something else. The formulation that works treats masking as the corruption:

Forward (training): take real text, mask a random fraction of
                    positions. At t=1 everything is masked; at t=0
                    nothing is.

Reverse (generation): start with every position masked, and at each
                    step predict tokens for all masked positions at
                    once, then commit the most confident ones and
                    re-mask the rest for the next step.

The model is a bidirectional transformer — no causal mask — so every position sees every other at every step. It is closer in shape to a BERT-style encoder run repeatedly than to a decoder run once per token.

The number of steps is chosen at inference. Fewer steps means more positions committed per step, which means more tokens decided without seeing each other’s final values — the quality dial.

What a step actually computes

state: a sequence of 512 positions, some tokens, some [MASK]

one step:
  1. forward pass over all 512 positions   (bidirectional)
  2. a distribution over the vocabulary at every masked position
  3. sample or take argmax at each
  4. keep the k most confident; re-mask the rest
  5. repeat

after K steps, all positions are filled.

Step 4 is where most of the design work sits. Committing purely by confidence tends to fill in easy, high-frequency tokens first and leave the load-bearing ones for later; committing left to right recovers fluency and gives back much of the parallelism. Real systems use schedules somewhere in between, and the schedule matters as much as the model.

Training, and the supervision it gives up

For each training example:
  1. sample a masking level t uniformly in (0, 1]
  2. mask each token independently with probability t
  3. predict the masked tokens from the unmasked ones
  4. cross-entropy on masked positions only, weighted by 1/t

The 1/t weight stops lightly masked examples, which are easy and
plentiful in expectation, from dominating the gradient.

One forward pass, one masking level, no iteration during training. The many-step procedure exists only at generation time, which is the same arrangement as image diffusion.

There is a cost in this objective that the speed discussion tends to skip. An autoregressive model gets a prediction target at every position of every example — a 1,000-token document yields 1,000 supervised predictions from one pass. A masked model gets targets only at the masked positions, which in expectation over a uniform t is half of them.

Per token of training data, the diffusion objective therefore extracts roughly half the supervision signal. Where data is the binding constraint rather than compute, that is a real disadvantage, and it is the same structural criticism that applies to masked language modelling generally. It is one concrete reason to be cautious about assuming equal-compute parity.

The latency argument, as arithmetic

Generate 512 tokens, and count two different things: sequential steps, and total token-positions processed.

AUTOREGRESSIVE, with a KV cache
  sequential steps      = 512
  positions per step    = 1 new token attending to the cache
  total new positions   = 512
  each step reads the full weights: memory-bound, badly under-utilised
                                    at batch size 1

DIFFUSION, 32 steps, sequence length 512
  sequential steps      = 32
  positions per step    = 512   (no cache: every position may change)
  total positions       = 32 * 512 = 16,384

  sequential steps: 16x fewer
  arithmetic:       32x more

That is the trade in one block. It is not a free speed-up; it is spending thirty-two times the arithmetic to remove sixteen times the sequential depth.

The reason that can be a good trade is what a GPU is like at batch size one. Single-stream decoding is memory-bandwidth-bound: the accelerator loads all the weights to compute one token and its arithmetic units are almost idle doing it. A diffusion step processes 512 positions with the same single weight load, so it uses hardware that was going to waste. Arithmetic you were not using is close to free; sequential steps never are.

Where the argument stops holding

Under load. Continuous batching already fixes the under-utilisation that the diffusion model is exploiting: with 64 concurrent requests, an autoregressive server loads the weights once and computes 64 tokens from that load, and the accelerator is no longer idle. At that point the extra arithmetic of a diffusion step is arithmetic you actually have to pay for.

Rough shape of it, per unit of work:

  batch 1:   AR wastes most of its arithmetic capacity
             -> diffusion's extra FLOPs are nearly free
             -> the 16x reduction in steps shows up as real latency

  batch 64:  AR is well utilised
             -> diffusion's 32x arithmetic is 32x arithmetic
             -> throughput per accelerator can favour AR

Three further costs are structural rather than situational:

• No KV cache reuse across steps. Every step may change any position, so the keys and values from the previous step are stale. The single largest optimisation in autoregressive serving does not apply, and neither does prefix caching in its usual form — though the prompt, being fixed, can be cached.
• The length has to be chosen up front. The sequence is a fixed-size canvas. An autoregressive model stops when it decides to; a diffusion model has to be told how much room to fill, and padding is wasted work.
• Streaming semantics differ. Tokens do not become final left to right, so the familiar experience of watching an answer appear in order requires a schedule that gives back parallelism.
• The optimisation stack does not transfer. Speculative decoding, which is the other main answer to sequential depth, assumes autoregressive verification and does not apply here.

Block decoding: the practical middle ground

The pure form is what gets quoted and it is not what most systems run. Semi-autoregressive block decoding sits between the two extremes and recovers most of what the pure form gives up:

Split the output into blocks of, say, 32 tokens.

for each block, left to right:
    denoise all 32 positions in parallel over K steps
    commit the block
    its keys and values are now FIXED -> cacheable

512 tokens, blocks of 32, 8 steps per block:
  sequential steps = 16 blocks * 8 = 128   (vs 512 autoregressive)
  work per step    = 32 positions, not 512
  KV cache         = valid for every completed block

The gains are real and modest instead of dramatic: a fourfold reduction in sequential steps rather than sixteenfold, in exchange for keeping a usable KV cache for everything already committed, keeping left-to-right streaming, and keeping the ability to stop when the model is finished rather than filling a fixed canvas.

Within a block, all the diffusion properties still hold — positions are decided jointly, and a token can be reconsidered before the block is committed. It is the sensible engineering answer, and it is worth knowing that it exists before comparing a headline steps-per-answer figure against an autoregressive baseline.

The properties that are not about speed

• Infilling is native. Fix the first and last paragraphs and generate the middle; the bidirectional model treats those as just more unmasked positions. An autoregressive model needs a special training format to do this.
• Revision is possible. A position committed early can be re-masked and reconsidered. Autoregressive generation cannot revisit a token without restarting, which is the mechanical reason a model that starts an answer badly tends to continue badly.
• Global constraints are easier. Structure that spans the whole output — a fixed schema, a required ending, a length — can be enforced against a whole-sequence view rather than token by token.

Honest status

Commercial diffusion language models exist and are marketed primarily on speed; open masked-diffusion models have been released at moderate scale, with LLaDA (2025) the most cited. The direction is active and well funded.