Inpainting and Outpainting: Masks Done Right

Almost every inpainting problem is one of two things: a mask edge that the latent grid cannot represent, or a masked region too small to have enough latent cells to work with. Both are consequences of the eightfold downsample, both are arithmetic, and both have a fix.

The short answer

Inpainting regenerates the masked region while conditioning on the rest. Outpainting is the same operation with the mask covering new canvas outside the original image. The pixels you keep are genuinely kept — the model never touches them — but the boundary between kept and generated is where all the difficulty lives, because the model works at one eighth of the resolution your mask was drawn at.

Two mechanisms, and how to tell which you have

There are two ways to inpaint, and knowing which one your pipeline is doing changes what the parameters mean.

Latent blending with an ordinary model

At every denoising step, the known region of the latent is overwritten with the original image noised to the current level, so the model only gets to decide the masked area. It works with any checkpoint and needs no special weights.

Its weakness is that the model was never trained on this task. It sees a latent whose unmasked part is a real noised image and whose masked part is its own evolving guess, and it has to make them agree using only attention across the latent. Results are usually acceptable for texture and poor for anything that must continue a structure across the boundary.

An inpainting-specific checkpoint

Some models are trained for the task, with extra input channels: the usual latent channels, plus the encoded masked image, plus the mask itself downsampled to latent resolution. The network is explicitly told what is known and what is to be filled, and it was trained on examples of exactly that.

A common inpainting UNet input layout, at 4 latent channels:

    4  noisy latent
    4  encoded masked image (the known pixels, VAE-encoded)
    1  the mask, downsampled to latent resolution
   ---
    9  input channels

A checkpoint with 9 input channels is an inpainting model. One with 4 is a
base model that your pipeline is masking by blending. If a pipeline errors
with a channel-count mismatch, that is the two being crossed.

Inpainting checkpoints handle structural continuation far better and are the right choice whenever you have one for your base model.

Your mask is quantised to eight pixels

The mask is applied in latent space, so it must be downsampled by the autoencoder’s factor. At f = 8:

Image mask:   1024 × 1024 pixels, drawn with a soft brush
Latent mask:   128 ×  128 cells

    1 latent cell = 8 × 8 pixels

Consequences that follow directly:

  • A mask 4 pixels wide covers half of one cell. After downsampling it is
    either present or absent depending on the rounding rule — often absent.
  • A feather of 3 pixels is sub-cell and does nothing. The transition you
    drew does not survive to the latent.
  • A feather of 24 pixels is 3 latent cells wide, which is a real gradient
    the process can act on.
  • The finest boundary the process can place is one cell, i.e. 8 pixels.
    Anything finer must be done by compositing in pixel space afterwards.

So the rule for feathering is not aesthetic, it is arithmetic: a feather smaller than f pixels is invisible to the model, and a useful feather is several times f. Sixteen to thirty-two pixels at f = 8 is the working range for most edits.

The second rule follows from the same fact. Even a perfect latent-space result must be decoded, and the decoder operates on the whole latent, so the pixels outside your mask come back very slightly changed — usually a small colour shift. The fix is to composite the result over the original in pixel space using the feathered mask, so the untouched regions are the original bytes rather than a re-decode of them.

The capacity problem, and crop-and-resize

This is the single largest quality lever in inpainting and the least known.

You want to fix a face in a 1024×1024 image. The face occupies
128 × 128 pixels.

    Latent cells available to the model for that face:
        (128 / 8) × (128 / 8)  =  16 × 16  =  256 cells

Now crop a padded box around the face, resize the crop to the model's
native 1024×1024, generate, and paste back:

    Latent cells available:
        (1024 / 8) × (1024 / 8)  =  128 × 128  =  16,384 cells

    Ratio: 64× more capacity for exactly the same subject.

That is the whole reason “only masked” or “inpaint at full resolution” modes exist, and it is why a face fixed with a whole-image inpaint looks smeared while the same fix done on a crop looks sharp. The model is not worse at faces in large images; it is being given 256 cells instead of 16,384.

1. Compute the bounding box of the mask.
2. Pad it — 25 to 50 per cent on each side — so the model sees context, not just the hole. Too little padding and it has nothing to match against; too much and you are back to the capacity problem.
3. Expand the box to the model’s aspect ratio and resize it to a native resolution. Which resolutions are native is covered in aspect ratio and training buckets.
4. Generate on the crop, with the mask cropped and resized the same way.
5. Resize the result back to the box’s original size and composite it over the untouched original using the feathered mask.

The comparison harness

This script produces the before-and-after. It holds the seed, the prompt and the mask fixed, varies exactly one parameter, and writes every output to disk so the comparison is controlled. Run it on your own image and your own model; the point is that the difference you see is attributable to the parameter and to nothing else.

# inpaint_compare.py — one parameter varied, everything else pinned.
# Requires: diffusers, torch, pillow, numpy.
# Pipeline and argument names should be checked against your installed
# version of diffusers.

import torch, numpy as np
from PIL import Image, ImageFilter
from diffusers import AutoPipelineForInpainting

MODEL  = "<your inpainting checkpoint>"
IMAGE  = "photo.png"
MASK   = "mask.png"          # white = regenerate, black = keep
PROMPT = "a wooden bookshelf against the wall"
SEED   = 4242
STEPS  = 30
CFG    = 6.0

pipe = AutoPipelineForInpainting.from_pretrained(
    MODEL, torch_dtype=torch.float16).to("cuda")

base = Image.open(IMAGE).convert("RGB")
raw  = Image.open(MASK).convert("L")

def feather(mask, radius):
    return mask if radius == 0 else mask.filter(ImageFilter.GaussianBlur(radius))

def run(mask, strength, tag):
    g = torch.Generator(device="cpu").manual_seed(SEED)
    out = pipe(prompt=PROMPT, image=base, mask_image=mask,
               num_inference_steps=STEPS, guidance_scale=CFG,
               strength=strength, generator=g).images[0]
    out = out.resize(base.size)
    # Composite in pixel space so untouched areas are the ORIGINAL bytes,
    # not a VAE round-trip of them.
    final = Image.composite(out, base, mask)
    final.save(f"out_{tag}.png")
    return final

# Experiment 1: feather radius. 0 and 4 are below the 8-pixel latent cell.
for r in (0, 4, 8, 16, 32, 64):
    run(feather(raw, r), 1.0, f"feather{r}")

# Experiment 2: strength, at a fixed sensible feather.
m = feather(raw, 16)
for s in (0.4, 0.6, 0.8, 1.0):
    run(m, s, f"strength{s}")

# Experiment 3: the crop-and-resize win. Compare out_feather16.png with this.
box = raw.getbbox()
pad = int(0.35 * max(box[2] - box[0], box[3] - box[1]))
crop_box = (max(0, box[0] - pad), max(0, box[1] - pad),
            min(base.width,  box[2] + pad), min(base.height, box[3] + pad))
c_img  = base.crop(crop_box).resize((1024, 1024), Image.LANCZOS)
c_mask = feather(raw.crop(crop_box).resize((1024, 1024), Image.LANCZOS), 16)
g = torch.Generator(device="cpu").manual_seed(SEED)
c_out = pipe(prompt=PROMPT, image=c_img, mask_image=c_mask,
             num_inference_steps=STEPS, guidance_scale=CFG,
             strength=1.0, generator=g).images[0]
merged = base.copy()
merged.paste(c_out.resize((crop_box[2]-crop_box[0], crop_box[3]-crop_box[1])),
             crop_box, feather(raw, 16).crop(crop_box))
merged.save("out_cropresize.png")

print("Open out_feather0.png next to out_feather16.png: that is the feather.")
print("Open out_feather16.png next to out_cropresize.png: that is capacity.")

The last two lines are the point of the script. Feathering fixes the seam; crop-and-resize fixes the detail. They are different problems and people routinely try to solve the second with the first.

Outpainting: extend in overlapping passes

Outpainting is inpainting where the mask covers canvas that did not exist. Three things make it harder than it looks.

1. Do not fill new canvas with grey. The area under the mask is encoded and given to the model as context in most pipelines. Flat grey is a strong signal that says “flat grey region”. Fill it by mirroring or by heavily blurring the adjacent edge, so the encoded context is at least statistically plausible.
2. Include a strip of the original in the mask. Masking only the new area leaves a hard boundary at exactly the original edge. Extend the mask 32 to 64 pixels into the existing image so the model can blend across it, and feather that inner edge.
3. Extend in steps, not in one jump. A single pass that triples the canvas gives the model a mostly-empty frame with a small island of context, and the result drifts into something unrelated. Extend by no more than roughly a quarter to a half of the current dimension per pass, and re-run.
4. Stay inside a native resolution each pass. Each pass should be a generation at a size the model was trained for, which usually means cropping a working window around the frontier rather than generating the whole growing canvas. This is the same capacity argument as above.
5. Expect perspective drift. Nothing enforces a consistent camera across passes. Long horizontal extensions accumulate a slowly changing horizon line, and the fix is structural conditioning with a depth or line signal, not a better prompt. See structural conditioning.

The five failures and their causes