How Diffusion Models Generate an Image

A diffusion model does not remove noise from an image. It repeatedly estimates all of the noise present in a tensor, takes a small step in the direction that estimate implies, and does it again. The difference between those two descriptions explains almost everything people find surprising about the results.

The short answer

Generation starts with a tensor of pure Gaussian noise. A neural network is called once per step, typically twenty to fifty times in total. Each call takes the current tensor and the current noise level and returns an estimate of the noise it contains. A sampler uses that estimate to produce a slightly less noisy tensor, and the loop repeats. When the noise level reaches zero, the tensor is the image — or, in a latent model, is decoded into one.

Two things follow immediately. Because the network is called once per step and each step depends on the last, generation time is linear in step count and cannot be parallelised across steps. And because the network only ever produces an estimate, the quality ceiling is set by the network, not by how many steps you run.

The forward process is the training signal

The model is trained by being shown the destruction it will later have to undo. Take a real image x0, pick a random noise level t, sample Gaussian noise ε, and build a corrupted version:

x_t = sqrt(alpha_bar_t) * x0  +  sqrt(1 - alpha_bar_t) * eps

alpha_bar_t  runs from ~1 at t=0 (almost pure image)
             to   ~0 at t=T (almost pure noise)

That single line is the whole forward process. Note what it is not: it is not an iterative corruption you have to simulate. Any noise level can be produced from a clean image in one operation, which is why training is cheap per sample and why the model sees the entire range of corruption levels during training rather than a trajectory.

The network is then given x_t and t and asked to predict the ε that was added. The loss is mean squared error between the predicted noise and the actual noise. That is the entire training objective for the classic formulation.

What the network actually predicts

The output of a forward pass has the same shape as its input. If the latent is 128×128×4, the prediction is 128×128×4. It is a full-size noise estimate, not a patch, not a correction, not a token.

The useful consequence is that you can algebraically rearrange the forward equation to recover an implied clean image at any point in the trajectory:

x0_hat = ( x_t - sqrt(1 - alpha_bar_t) * eps_hat ) / sqrt(alpha_bar_t)

This x0_hat is what generation UIs show as a live preview. At step 2 of 30 it is a blurred arrangement of colour masses with roughly the right composition. At step 25 it is nearly the final image. The model is, at every step, guessing at the entire finished picture; the sampler only ever moves part of the way there.

Three prediction parameterisations are in common use and they are equivalent up to a change of variables: ε-prediction (predict the noise, the original formulation), x0-prediction (predict the clean image directly), and v-prediction (predict a mixture of the two that has better-conditioned targets at both ends of the schedule). A checkpoint is trained for one of them and a sampler must be told which, which is one of the ways a mismatched configuration produces grey mush rather than an error message.

What one sampling step changes

Given the noise estimate, the sampler moves from noise level t to a lower level t-1. Deterministic samplers do only that. Stochastic samplers subtract slightly more than the step requires and then add fresh noise back, which is why they never converge to a fixed image no matter how many steps you run. That distinction is covered in what the samplers actually change.

The size of that move is not uniform. The noise schedule is designed so that the early steps traverse a large amount of noise level and the late steps traverse very little, because the signal-to-noise ratio changes by many orders of magnitude across the trajectory. In practice this means the first three or four steps do most of the visible work, and the last ten adjust texture.

Why early steps decide composition

This is the part most explanations omit, and it is the one that makes the behaviour predictable.

Natural images have most of their energy at low spatial frequencies — large regions of similar colour — and comparatively little at high frequencies. Gaussian noise has equal energy at every frequency. So when you add a lot of noise to an image, the high-frequency content disappears beneath the noise floor first, while the low-frequency content is still detectable.

At a high noise level the only information the network can extract from x_t is the coarse layout: where the light areas are, where the dark ones are, the rough silhouette. So the early steps commit to composition, and once committed, the later steps have no mechanism to revise it — a step only ever refines the tensor it was given.

• Composition is decided in the first few steps. This is why the same seed with a different prompt often gives a similar arrangement, and why the seed feels like it controls layout.
• Late steps cannot fix structural errors. If step 5 committed to a blob where a hand should be, step 28 will fill that blob with plausible finger texture. That is one of the mechanisms behind the hands and text problem.
• Starting part-way through skips the structural phase. That is exactly what image-to-image does: it noises your input to a chosen level and enters the loop there, so everything decided above that noise level is inherited rather than generated.

Where the prompt enters

The prompt is encoded once, by a separate text encoder, into a sequence of embedding vectors. Those vectors are supplied to the denoising network at every step, usually through cross-attention layers in which the image positions attend to the text tokens.

Nothing about this forces the output to match the prompt. The network learned a conditional distribution, and a sample from it is only as prompt-adherent as the training made it. That is why classifier-free guidance exists: it is a sampling-time trick that exaggerates the difference between the conditioned and unconditioned predictions to push the sample further towards the prompt. It costs a second forward pass per step, which is why guidance is the single largest multiplier on generation cost after step count.

Flow matching: the same skeleton, restated

Recent image and video models are frequently described as flow matching or rectified flow models rather than diffusion models. The reframing is real but the skeleton is the same: instead of a noise schedule derived from a stochastic corruption process, you define a straight path from noise to data, and train the network to predict the velocity along that path. Sampling is then integrating that velocity field.

The practical consequences are that the trajectory is closer to a straight line, which makes low step counts work better, and that the noise schedule stops being a design headache. Everything on this page about frequency ordering, guidance, conditioning and the linear cost in step count carries over unchanged, because none of it depended on which corruption process was chosen.