How Video Models Keep Frames Consistent

Frame consistency is not a post-process. It comes from generating every frame in one joint denoising process, with attention connecting positions across time. That design decision is also what makes clips short, and the reason is arithmetic that can be worked in ten lines.

The short answer

A video model denoises a tensor with a time axis. Instead of a latent of shape (channels, height, width) it works on (channels, frames, height, width), and the network contains layers that connect the same spatial position across different frames. Consistency is therefore a property of the sample, not something enforced afterwards.

Everything else — compressing time as well as space, conditioning on a first frame, extending a clip in chunks — is engineering around the cost of that joint process.

Why independent frames flicker

Generate 24 frames independently with the same prompt and 24 different seeds, and you get 24 unrelated images. Use the same seed for all of them and you get 24 nearly identical images with no motion. Neither is video.

The reason is that a diffusion model samples from a distribution. Nothing in an independent sample knows about its neighbours, so every decision the model makes freely — the exact pattern of a texture, the precise shade of a surface, the fold of a garment — is redrawn every frame. The result is the characteristic boiling: shapes hold, surfaces crawl.

Interpolating between two generated keyframes does not fix it either. Interpolation produces a smooth transition between two samples, and if the two samples disagree about what an object is, the smooth transition is a morph rather than motion.

Temporal layers on an image model

The first workable approach, and still a good way to understand the problem: take a trained image model, insert new layers that operate along the time axis, and train those on video while keeping the image weights frozen or lightly tuned.

The latent, with a time axis:

    z  shape  (batch, channels, frames, height, width)

Spatial attention  — the layer the image model already had.
    Reshape so each frame is its own sequence of positions.
    Frames do not see each other. Cost per frame is unchanged.

Temporal attention — the new layer.
    Reshape so each spatial position is a sequence over frames.
    Position (x, y) in frame 3 attends to position (x, y) in every
    other frame. Quadratic in frame count, but the sequences are
    short, so it is cheap relative to spatial attention.

This factorisation is what makes the approach affordable: full attention over all positions in all frames at once is quadratic in the product, while doing spatial and temporal attention separately is quadratic in each independently.

Its weakness is equally structural. Temporal attention at a fixed spatial position handles a scene where things stay roughly where they are. It handles large motion poorly, because the corresponding content has moved to a different position and the layer has no path to it except through the spatial layers. Objects that move quickly across frame are where this design visibly struggles.

Compressing time as well as space

The more capable design compresses the time axis too. Instead of an image autoencoder applied per frame, a video autoencoder takes a block of frames and produces a smaller block of latent frames.

Image autoencoder applied per frame, f = 8:
    120 frames at 512×512  →  120 × 64 × 64 latent positions

Video autoencoder, spatial f = 8 and temporal f = 4:
    120 frames at 512×512  →   30 × 64 × 64 latent positions

    A 4× reduction along the time axis before the denoiser sees anything.

With time compressed, the denoiser can afford full attention over the whole spatiotemporal token grid rather than the factorised approximation — every position in every latent frame attending to every other. That is what buys global coherence: an object leaving frame and returning has a path through attention to where it was.

The cost is that the autoencoder now has to reconstruct motion, and it is where temporal artefacts originate: ghosting on fast movement, smearing at cut boundaries, and a characteristic softness during rapid motion. The reconstruction-ceiling argument from latent diffusion applies, with an extra axis.

The arithmetic that limits clip length

This is the constraint that decides everything about how video generation is deployed, and it is derivable.

Assumptions, all stated:
  A1  output 512 × 512, 24 fps
  A2  spatial downsample f = 8, temporal downsample 4
  A3  transformer patch size 2 on the spatial axes, 1 on time
  A4  full self-attention over the whole spatiotemporal grid

Token count for a clip of D seconds:

    frames        = 24 × D
    latent frames = 24 × D / 4          = 6 × D
    spatial tokens per latent frame = (512/8/2)² = 32² = 1,024
    total tokens  N = 6 × D × 1,024     = 6,144 × D

    D = 1 s   →  N =   6,144
    D = 2 s   →  N =  12,288
    D = 5 s   →  N =  30,720
    D = 10 s  →  N =  61,440

Compare with one 512×512 image at the same settings:  N = 1,024.

Attention work is proportional to N²:

    one image   1,024²    =  1.05 × 10^6      1×
    1 s clip    6,144²    =  3.77 × 10^7     36×
    2 s clip   12,288²    =  1.51 × 10^8    144×
    5 s clip   30,720²    =  9.44 × 10^8    900×
   10 s clip   61,440²    =  3.78 × 10^9  3,600×

Dense work is proportional to N, so it grows only 6×, 12×, 30×, 60×.
Attention is what makes long clips expensive, and it grows with the
SQUARE of duration.

Doubling clip duration quadruples the attention cost. That single line explains most of what you observe about video generation: why clips are measured in seconds, why higher resolution is traded against length, why longer output is produced by joining chunks rather than in one pass, and why every architecture in this area is in some way an attempt to avoid paying the quadratic term.

Approaches that attack it include the factorised spatial-temporal attention above, windowed attention over a limited time neighbourhood, hierarchical generation of keyframes followed by in-between frames, and heavier temporal compression in the autoencoder. Each trades a different kind of coherence for the saving.

What each conditioning mode fixes

The named modes in a video interface are not features so much as answers to “which part of the output is decided by you rather than sampled”. Each removes a specific source of variance, and knowing which one removes which is how you pick.

The ordering is the practical takeaway. Given the cost derived above, the cheapest way to get a clip you want is almost always to move as much of the decision as possible into the still image, where iteration costs a fraction as much, and then use image-to-video for the part that genuinely needs a video model. All of the image-side control in this cluster — structural conditioning, masked editing, identity conditioning — is available on that first frame and mostly is not available mid-clip.

Extending a clip, and drift

Longer output is generally produced by generating a chunk, then generating the next conditioned on the last frames of the previous one, and joining. It works, and it has a specific failure.

• Error accumulates. Each chunk conditions on a generated frame rather than a real one, and any drift in colour, lighting, identity or style compounds. Over enough chunks the scene becomes a different scene, gradually enough that no single join looks wrong.
• Overlap helps and costs. Generating chunks that overlap by several frames and blending gives smoother joins at the price of regenerating those frames.
• Global conditioning helps more. Conditioning every chunk on the same first frame, or on the same reference image, anchors appearance and limits drift better than chaining alone.
• Nothing here enforces continuity of events. A chunked generation has no representation of what happened; it has the last few frames. An object that left the frame two chunks ago does not exist.

What remains hard, and why

These are structural rather than a matter of scale, which is why they are worth naming separately from anything about current capability.

• Physics is statistical, not enforced. The model learned what falling objects look like. There is no simulator and no conservation law. Plausible-looking motion that violates physics is not a bug; it is a sample from a distribution fitted to appearances.
• Object permanence is bounded by the attention window. If a thing is out of view for longer than the model can attend across, its return is a new sample.
• Counting and text inherit every image-model problem. All the causes in text and hands apply per frame, and now have to be consistent across frames as well.
• Evaluation is unsettled. There is no equivalent of glancing at an image. Temporal artefacts require watching, at speed, repeatedly, and automated metrics for temporal coherence are not well established.