ControlNet, Depth Maps and Structural Conditioning

ControlNet is a trainable copy of half the denoising network, connected back to the frozen original through convolutions initialised to zero. Because those connections start at zero, the model at the beginning of training is bit-identical to the model before it — which is the trick that makes the whole approach work on a modest dataset.

The short answer

You want to keep a model’s knowledge but constrain where things go. Fine-tuning the model on paired data risks destroying what it knows. So instead: freeze the original, clone its encoder into a trainable branch, feed the conditioning image into that branch, and add the branch’s outputs back into the frozen network’s skip connections.

The approach was published by Zhang and co-authors in 2023 as Adding Conditional Control to Text-to-Image Diffusion Models, and the name has since become generic for the whole family of structural conditioning adapters.

The architecture, and the zero convolutions

1. Freeze the original network. Its weights never change, so it cannot forget anything.
2. Clone the encoder half and the middle block. This copy is trainable and starts from the frozen model’s weights, so it begins with the same feature extractors rather than from random initialisation.
3. Encode the conditioning image. The edge map, depth map or pose skeleton is passed through a small convolutional stack that brings it to the latent’s spatial resolution, and added to the copy’s input.
4. Connect the copy back through zero convolutions. 1×1 convolutions whose weights and biases are initialised to zero sit on both the input and the output of the trainable branch.
5. Train. On step zero the branch contributes exactly zero, so the composite model produces precisely what the frozen model would have. Training then grows the contribution from nothing.

The zero initialisation is the part worth understanding. A randomly initialised branch would inject noise into a working model, and the first thousand training steps would be spent undoing the damage — on a small dataset, that damage may never be fully undone. Starting from an exact no-op means every gradient the branch receives is about the conditioning signal and nothing else.

A zero-initialised layer has zero gradient with respect to its own weights only if its input is also zero. Here the input is the non-zero activations of the copied trunk, so the gradient is non-zero and the layer learns. This is why the trick does not simply get stuck.

What each conditioning signal actually constrains

The useful mental model is: a conditioning image is a per-pixel instruction at latent resolution. It says something about where, and nothing about what. Different signals say different things about where.

The choice between them is mostly a choice of how much you are delegating. Canny hands the model a stencil. A pose skeleton hands it a stick figure and asks it to invent everything else.

The preprocessor is half the result

The conditioning image is produced by a preprocessor — an edge detector, a monocular depth estimator, a pose estimator — and the quality of that output bounds the quality of the control. This is the step people skip, and it is where most disappointing results come from.

• Canny thresholds change everything. Low thresholds produce a dense map full of texture edges, and the model will faithfully reproduce noise you did not want. High thresholds drop structure you needed. Look at the edge map before you generate; it takes two seconds and it tells you what the model is going to be told.
• Depth estimators disagree about range. Different estimators produce different near-to-far mappings, and some output inverse depth. A map that is flat across most of its range gives the model almost no signal even though it looks like a depth map.
• Pose estimators fail silently on partial bodies. A missing wrist keypoint means the model gets no instruction there and invents one, which is usually the joint that ends up wrong.
• Resolution matters. The conditioning image is resized to the generation resolution. A 512-pixel edge map used for a 1024-pixel generation has had its detail interpolated, and the result is soft control.

Strength, start and end

Three parameters govern how much the branch is allowed to say, and they act on different axes. Names vary between implementations — check them against the documentation of the library you are using — but the concepts are consistent.

The reason ending control early works is the frequency ordering described in how diffusion works: structure is committed in the early steps, so control applied after that point is constraining decisions that have already been made, while suppressing the free texture generation you want at the end.

Stacking multiple branches is additive. Two branches at scale 1.0 each contribute their full trained residual, and the sum frequently overwhelms the frozen model. Halve each when using two, and expect to go lower again with three.

What it costs per step

The branch is a copy of the encoder and the middle block, and it runs at every step at which control is active. It is not free and it is not a doubling.

Per denoising step, with control active:

    frozen network       encoder + middle + decoder
    control branch       encoder + middle   (a copy)

So the extra work is the encoder-and-middle share of one forward pass.
In a symmetric encoder-decoder that share is on the order of half, but it
depends on the architecture and there is no substitute for measuring it.

Interaction with guidance: the control branch runs on both the conditional
and unconditional passes in most implementations, so the increase applies
to both. With guidance on and control on, expect roughly:

    2 passes/step × (1 + branch_fraction)

Setting guidance_end to 0.6 removes the branch from 40% of steps.

Adapters that compute their conditioning features once and inject them at every step, rather than running a full branch per step, exist for exactly this reason — they trade control fidelity for a much smaller per-step cost. If a control-conditioned generation is too slow, that class of adapter is the first alternative to look at.

When something else is the right tool

• You want a specific object changed, not the whole composition constrained. Use a mask. See inpainting and outpainting.
• You want the same subject across many images. Structural conditioning fixes geometry, not identity. Two images with the same pose skeleton can be two different people. See keeping a character consistent.
• You want a style. A style is a property of every pixel, not a spatial arrangement. A LoRA is the right tool and a control branch is not.
• You want to keep most of an existing image. Image-to-image at low strength preserves the actual pixels, which structural conditioning never does — it reproduces the geometry with entirely new content.