Latent Diffusion: Why It Runs on a Laptop

Latent diffusion is one idea: do not run the diffusion process on pixels, run it on a compressed representation, and use a separately trained autoencoder to go in and out. The compression factor is derivable in one line, and it cuts the quadratic part of the compute by a factor of several thousand. That is the whole reason megapixel-scale generation runs on consumer hardware.

The short answer

A convolutional autoencoder is trained first, on its own, to reconstruct images through a bottleneck that reduces each spatial dimension by a factor f — almost always 8 — while widening the channel dimension from 3 to some small number c. Its encoder and decoder are then frozen.

The diffusion model is trained entirely inside that bottleneck. It never sees a pixel. At generation time the loop runs on a latent tensor, and the decoder is called exactly once at the end to turn the final latent into an image.

Deriving the compression factor

Take a 1024×1024 RGB image and a standard autoencoder with f = 8.

Pixel tensor:   1024 × 1024 × 3   = 3,145,728 values
Latent tensor:  (1024/8) × (1024/8) × c
             =  128 × 128 × c

  c = 4  →  128 × 128 × 4  =    65,536 values   →  48× compression
  c = 16 →  128 × 128 × 16 =   262,144 values   →  12× compression

General form:   compression = (f² × 3) / c

  f=8,  c=4   →  (64 × 3) / 4   = 48
  f=8,  c=16  →  (64 × 3) / 16  = 12
  f=16, c=16  →  (256 × 3) / 16 = 48

So the much-quoted “48× compression” is not a property of latent diffusion in general. It is what you get at f = 8 with four latent channels, and it drops to 12× the moment the channel count goes to sixteen. The trade that buys is covered below.

What that does to the compute

The compression ratio of the data is not the interesting number. The interesting number is what happens to the arithmetic, because the two dominant cost terms scale differently.

Let N be the number of spatial positions the denoising network processes. Every dense operation — convolutions, projections, the MLP in a transformer block — costs work proportional to N. Self-attention costs work proportional to N², because every position attends to every other.

Positions at 1024×1024 output, one position per pixel:
    N_pixel  = 1024 × 1024 = 1,048,576

Positions in an f=8 latent, with a transformer patch size of 2:
    latent   = 128 × 128
    N_latent = (128/2) × (128/2) = 64 × 64 = 4,096

Ratio:
    linear term     N_pixel  / N_latent  = 1,048,576 / 4,096   =    256×
    quadratic term  N_pixel² / N_latent² = 256²                = 65,536×

Stated in terms of the downsample factor alone, and setting the patch size aside: dense work falls by f² = 64, and attention work falls by f⁴ = 4,096. Even at these sizes the quadratic term is the one that decides whether the thing is buildable at all.

Put it the other way round to see what was actually bought. A pixel-space model with full self-attention over a megapixel image would need on the order of 10¹² attention interactions per layer per step; the latent model needs about 1.7×10⁷. Nobody chose latent diffusion because it was elegant. They chose it because the pixel-space version was not going to run.

The same scaling is why doubling output resolution costs more than four times as much, worked in full in the GPU-seconds page.

What the autoencoder is and what it costs

It is a variational autoencoder, usually referred to as the VAE, and it is trained with a combination of reconstruction loss, a perceptual loss and often an adversarial loss. The adversarial term is why decoded images look sharp rather than blurry: a plain reconstruction loss produces the mean of the plausible outputs, which is a blur.

Its cost profile is unlike the denoiser’s. The encoder and decoder run at full pixel resolution, so they are individually expensive per call — but they are called once each per generation rather than once per step.

Per generation, 30 steps with guidance:

  denoiser calls   = 30 steps × 2 (guidance)  = 60 full passes on the latent
  VAE decode       = 1 pass on the pixel grid
  VAE encode       = 1 pass, only for img2img / inpainting

So the decode is roughly 1/60th of the call count, but each call operates on
64× more spatial positions. Expect it to be a small but not negligible
fraction of wall-clock time — measure it rather than assuming it is free,
because at low step counts it stops being small.

That last point matters for step-distilled models. At four steps with no guidance there are four denoiser calls and one decode, and the decode can become a substantial share of the total. It is a common surprise when a four-step model turns out not to be seven times faster than a thirty-step one.

Four channels or sixteen

The channel count of the latent is the autoencoder’s information budget, and increasing it is the clearest quality lever in the whole stack.

The reconstruction ceiling is the concept worth carrying away. Encode an image and immediately decode it, with no diffusion involved at all. Whatever is lost in that round trip is lost for every generation that model will ever produce. It is a floor on error that no amount of steps, guidance or prompt work can get under.

What the autoencoder destroys

The round-trip test above is also the diagnostic. Run it on a photo containing the things people complain about and you can see which failures belong to the denoiser and which belong to the compressor.

• Small text. A letter with an 8-pixel cap height occupies one latent position at f = 8. There is no representation available in which that glyph is a shape. This is derived properly in why image models struggle with text.
• Fine repeating texture. Woven fabric, distant foliage, hair at small scale, halftone patterns. These sit at frequencies the encoder cannot keep and the decoder resynthesises approximately.
• Precise thin lines and grids. Architectural drawings, wireframes, one-pixel rules. They come back wobbly.
• Colour fidelity in flat regions. Some autoencoders introduce a slight shift or a low-amplitude pattern in large uniform areas, which is visible in skies and studio backdrops and is a common reason a generated image will not composite cleanly.

Why the autoencoder is trained separately

The two-stage arrangement — train the compressor first, freeze it, then train the generator inside it — is a design decision with consequences worth understanding, because it explains several things that otherwise look arbitrary.

• The two objectives conflict. An autoencoder wants the latent to be an efficient, faithful code. A diffusion model wants the latent to be a space where a smooth trajectory from noise to data exists. Training them jointly means one objective degrades the other, and in practice the reconstruction loses.
• The compressor is reusable. One autoencoder can serve many generative models, and it can be trained on far more data than any single generative run needs because it requires no captions at all. Images alone are enough.
• Freezing bounds the failure. With the compressor fixed, the reconstruction ceiling is a known constant. Every improvement to the generator moves the result towards that ceiling and never past it, which makes the ceiling a useful thing to have measured.
• It also means the two can be mismatched. Swapping in a different autoencoder is a change to the space the generator was trained in, and it is only safe when the replacement was deliberately trained to be compatible — same downsample factor, same channel count, same scaling factor, and a latent distribution close enough that the trajectory still lands somewhere the decoder understands. A mismatch produces images with a colour cast, a loss of contrast, or an overall texture that is subtly wrong everywhere, rather than an error.

That last point is the practical one. When a generation looks plausible but flat, washed out, or oddly grainy across the whole frame, and the prompt and the sampler are not to blame, the autoencoder is the component to check — and the round-trip test below distinguishes it from everything else in one run.

Checking your model’s numbers

Every figure on this page is a function of two values that are recorded in the model’s own configuration, so there is no reason to guess them.

# Read f and c from the model itself rather than from a blog post.
import torch
from diffusers import AutoencoderKL

vae = AutoencoderKL.from_pretrained("<your checkpoint>", subfolder="vae")
print("latent channels:", vae.config.latent_channels)
print("downsample f   :", 2 ** (len(vae.config.block_out_channels) - 1))
print("scaling factor :", vae.config.scaling_factor)

# The round-trip test: what this autoencoder costs you before any generation.
from PIL import Image
import numpy as np
img = Image.open("test.png").convert("RGB").resize((1024, 1024))
x = torch.from_numpy(np.asarray(img)).float().permute(2,0,1)[None] / 127.5 - 1.0
with torch.no_grad():
    z = vae.encode(x).latent_dist.mean
    y = vae.decode(z).sample
print("latent shape   :", tuple(z.shape))
print("round-trip RMSE:", float(((y - x) ** 2).mean().sqrt()))

# Attribute names in diffusers have changed across versions; check them
# against the version you have installed if any of these raise.

The scaling_factor in that output is worth knowing about. Raw latents from these encoders do not have unit variance, and the diffusion model was trained on latents multiplied by that constant. A pipeline that omits it produces images that are catastrophically wrong in a way that looks like a broken model rather than a missing multiplication.