Image Editing and Inpainting via API

Generating an image is a demo. Editing one is a product, because real use is iterative: remove the logo, extend the background to 16:9, change the shirt colour, keep everything else identical. The APIs for this look simple and the failures are concentrated in two places — the mask, and what repeated editing does to the pixels you did not touch.

Three kinds of edit

• Masked inpainting. You supply an image, a mask, and a prompt describing what the masked region should contain. The model regenerates inside the mask and is supposed to leave the rest alone. The classic operation: remove an object, replace a sky, fix a hand.
• Outpainting. The same operation with the mask outside the original bounds — pad the canvas, mask the new margin, and let the model extend the scene. This is how you turn a 1:1 asset into a 16:9 hero image without cropping the subject.
• Instruction editing. No mask. “Make the sign green.” The model decides the region itself. Far more pleasant to use, and the trade is that you have surrendered the guarantee that nothing else changed — because in general something else did.

The distinction that matters commercially is that last one. If a legal or brand requirement says the product photograph must be unaltered outside the edited region, instruction editing cannot give you that assurance and masked inpainting can, at least in principle.

The mask is where it goes wrong

Almost every first integration fails on the mask, and usually on the same three things.

Which channel carries it. Some APIs want a fully transparent hole in an RGBA PNG — the alpha channel is the mask and transparent means “edit here”. Others want a separate greyscale image where white means edit, and others where black means edit. Getting the polarity backwards produces a confident, complete, entirely wrong result: everything except your intended region is regenerated. If your first edit comes back with the subject replaced and the background intact, you have inverted the mask.

Dimensions must match exactly. A mask one pixel off from the source is a hard error on most APIs, and worse, a silently-resized mask on some — which shifts the edit region by a few pixels and produces a result that looks almost right.

Antialiasing is not free. A mask drawn with a soft-edged brush has intermediate alpha values, and APIs differ in whether they respect them or threshold them. Building the mask programmatically is the way to keep this predictable:

from PIL import Image, ImageDraw, ImageFilter

src = Image.open("photo.png").convert("RGBA")
w, h = src.size

# transparent hole = the region to regenerate
mask = Image.new("L", (w, h), 255)          # 255 = keep
ImageDraw.Draw(mask).rectangle([420, 180, 760, 520], fill=0)
mask = mask.filter(ImageFilter.GaussianBlur(6))   # feather the edge

edit = src.copy()
edit.putalpha(mask)
edit.save("edit_input.png")                  # dimensions unchanged

Seams, leakage and feathering

Latent-space models do not edit pixels; they edit a compressed representation in which one latent cell corresponds to a block of pixels — an 8 × 8 block is a common factor. Two consequences follow directly.

Your mask boundary is quantised to that grid, so a pixel-precise mask is not honoured pixel-precisely, and the edit bleeds a few pixels past the line you drew. Feather the mask by roughly that block size and the transition stops being a visible edge.

And the regenerated region is decoded independently of the surrounding pixels, so it can differ subtly in noise characteristics, film grain and colour temperature even when the content is perfect. On a flat studio background this is invisible; on a grainy photograph it produces a rectangle that a viewer notices without being able to say why. The practical fix is to composite the result back yourself using a blurred version of your own mask, so the original pixels are preserved except where you truly wanted change.

Round trips degrade the whole image

This is the one that ambushes people six weeks in. Each edit typically decodes and re-encodes the entire image through the model’s autoencoder, even the parts you masked as untouchable. That is lossy. One pass is imperceptible. Eight passes — which is a normal number for a user iterating on a design — accumulate into visible softening, drifting colours and disappearing fine texture.

Two defences, and you want both. Composite each edit back onto the original at full resolution rather than chaining the model’s output into the next request, so the untouched region is always the true original. And where the workflow permits, collect the user’s intended changes and apply them in one pass rather than eight — a single edit with three masked regions beats three sequential edits on both fidelity and cost.

Also mind the file format: saving intermediates as JPEG adds a second, independent generation loss on top of the autoencoder’s. Keep working copies in PNG or another lossless format and convert once at the end.

Resolution is the other thing that quietly drifts. Many editing endpoints operate at a fixed working size, so a 4000-pixel original comes back smaller than it went in, and a user who edits four times ends up with an asset they cannot print. If the output dimensions do not match the input dimensions, treat that as a finding rather than a detail: either upscale deliberately as a final step, or run the edit on a downscaled proxy and composite the result back onto the full-resolution original through the same mask, which preserves every pixel you did not intend to change.

A pipeline that survives contact with users

• Keep the original immutable. Every edit is original + a list of operations, never a chain of outputs. This makes undo trivial and stops accumulation.
• Store the mask with the result. When someone asks why a region changed, the mask is the answer. It is also what you need to re-run the edit against a higher-resolution original.
• Verify programmatically. After compositing, diff the result against the original outside the mask. Any non-zero difference there is a bug — an inverted mask, a resize, or an API that ignored the mask. This check costs nothing and catches the worst class of failure before a user sees it.
• Expect refusals mid-workflow. An edit request carries an image and a prompt, so both go through the safety filters. An edit that was fine yesterday can be refused today; the UI needs a path for that which is not a stack trace.
• Preserve provenance metadata. Where the vendor attaches C2PA or similar signed provenance to generated content, stripping it during your compositing step is a decision, not an accident. Make it deliberately.