Negative Prompts and How They Work Mechanically

There is no negative-prompt mechanism in a diffusion model. There is a text field that gets encoded and put where the empty string normally goes — in the unconditional branch of classifier-free guidance — and every property of negative prompts, including the surprising ones, falls out of that one substitution.

The short answer

Guidance evaluates the denoiser twice per step: once with your prompt, once with an empty prompt. The two predictions are combined by extrapolating away from the empty-prompt one. A negative prompt simply puts text in that second slot, so you are now extrapolating away from a specific prediction rather than a generic one.

It is not a filter, not a constraint, and not a post-generation check. It is a direction in noise space that the sampler is told to move against.

The slot it occupies

Standard classifier-free guidance, no negative prompt:

    eps_hat = eps("")        +  w · ( eps(prompt) − eps("") )

With a negative prompt, "" is replaced by the negative text:

    eps_hat = eps(negative)  +  w · ( eps(prompt) − eps(negative) )

Expanded into coefficients:

    eps_hat = w · eps(prompt)  +  (1 − w) · eps(negative)

The negative prompt is therefore weighted 1 − w, which is negative for any guidance scale above 1. That is the whole thing. Read off the coefficients at the usual settings, using the same table worked in the guidance page:

  w        weight on prompt     weight on negative prompt
-----------------------------------------------------------
  1.0            1.00                 0.00     no effect at all
  2.0            2.00                −1.00
  3.5            3.50                −2.50
  5.0            5.00                −4.00
  7.5            7.50                −6.50
 12.0           12.00               −11.00     very strong; expect damage

Why it does nothing at guidance scale 1

At w = 1 the coefficient on the negative prompt is exactly zero. Substitute it into the equation and see it disappear:

    eps_hat = 1 · eps(prompt)  +  (1 − 1) · eps(negative)
            = eps(prompt)

The negative prompt term is multiplied by zero. Not weakened. Removed.

Most implementations go further and skip the second forward pass entirely when the scale is 1, since it would be multiplied out anyway. So the negative prompt field is not merely ineffective at that setting — the text is never encoded and the pass never runs.

This has a practical corollary that catches people constantly: the strength of a negative prompt is not a separate dial. It is w − 1. Raising the guidance scale strengthens the negative prompt at the same time as it strengthens the positive one, and lowering the guidance scale to fix oversaturation weakens the negative prompt as a side effect.

Why it is free

The second forward pass already existed. Guidance requires it whether the second prompt is your negative text or the empty string, so filling the field costs one extra text-encoder call — a few milliseconds, once, not per step — and nothing else.

That is why negative prompts spread as a habit. They are the only quality lever in a generation UI with no compute cost attached, which also explains why enormous boilerplate negative prompts became normal: nobody was paying for them by the token.

The corollary is that on models with no second pass — guidance-distilled checkpoints, and models that were trained without conditioning dropout — there is nowhere for the negative prompt to go. A UI may still show the field and silently ignore it, which is a documentation problem rather than a bug in the model.

Why it fails to remove things

“I put hat in the negative prompt and it still gave me a hat.” The mechanism explains why this is expected rather than broken.

• It is a direction, not a constraint. The sampler moves against a vector at every step. Nothing checks the result. There is no stage at which the image is examined for hats.
• Nouns in the negative are not object-level. The text encoder produces a sequence of embeddings for the whole negative prompt, and the model attends to all of it. Putting “hat” there pushes away from the model’s general notion of hat-related imagery, which is entangled with heads, portraits, and whatever else co-occurs with hats in training data.
• The positive prompt usually wins. The coefficient on the positive prompt is w and on the negative is 1 − w, and the positive prompt is also the thing the model was conditioned to follow. If the composition needs a hat, guidance of 7.5 against it is a nudge.
• Long negatives dilute each other. One text encoding represents the whole negative string. Adding a twentieth term does not add a twentieth constraint; it changes one embedding slightly. Encoders with a 77-token limit will also silently truncate a long negative prompt, which is why the end of a boilerplate list often does nothing at all.

For actually removing an object, the reliable route is spatial rather than textual: mask it and regenerate that region, as covered in inpainting and outpainting, or constrain the composition with structural conditioning.

Writing one that does something

1. Start empty and add only what you observe. A negative prompt copied from a forum is a set of directions tuned for somebody else’s model at somebody else’s guidance scale. On a modern checkpoint it frequently makes things worse.
2. Prefer quality and style words to object words. Terms describing a rendering quality — blur, low resolution, compression artefacts, a medium you do not want — describe a direction in image space, which is what this mechanism can act on. Object nouns describe content, which it cannot reliably suppress.
3. Keep it inside the token limit. If your text encoder truncates at 77 tokens, that is roughly 50 to 60 words. Anything past it is discarded silently.
4. Test with the seed fixed. Generate with and without the negative prompt at the same seed, sampler and step count. Any other comparison is noise. See seeds and determinism.
5. Re-tune it when you change the guidance scale. The negative prompt’s strength is w − 1. A negative prompt tuned at guidance 7.5 is running at less than half strength at guidance 3.5.

Beyond the single negative

Two extensions exist and both are visible in the equation.

Multiple weighted conditions. Nothing restricts the combination to two terms. Implementations exist that evaluate several conditionings and combine them with independent weights, which is how per-concept strengths are offered in some tools. The cost is one forward pass per condition, so a three-way combination triples the per-step compute rather than doubling it.

Perpendicular negative guidance. A known failure of the plain formulation is that the negative direction often overlaps with the positive one — negating “blurry” also pushes against parts of what you asked for. The fix is to project the negative direction onto the component orthogonal to the positive one and use only that, so the negative term cannot cancel anything the prompt asked for. It costs the same forward passes and only changes how the vectors are combined.

Both are worth knowing about mainly as a way to read what a tool is doing. If a UI offers per-term negative weights, it is doing something other than the two-term equation above, and its numbers will not match this page’s coefficients.