Prompting for Images vs Prompting for Text

People who are excellent at prompting language models often write bad image prompts, and the reason is not a lack of practice. In a classic diffusion stack your words never reach anything that reasons about them: they are compressed into a conditioning vector by a small text encoder, and that vector steers a denoiser. Almost every difference in technique follows from this one fact.

Your prompt is read by different machinery

text model
  prompt -> tokens -> 70+ layers of attention -> reasoning over
            your instruction -> tokens out

diffusion image model
  prompt -> tokens -> small text encoder -> conditioning vector
         -> steers 20-50 denoising steps -> pixels
                    ^
                    no instruction following happens here

The encoder was trained to place captions near their images in a shared space — a CLIP-style objective. It is a similarity machine. Newer systems improved matters considerably by adding a large language model as an additional text encoder, which is exactly why instruction following in recent image models is so much better than it was, and why advice written for the 2022 generation now reads as superstition. The two regimes coexist: an open-weight model you self-host may still be running the small encoder, while a current hosted API may be reading your prompt with something much closer to a language model.

Why “no elephants” produces elephants

Negation requires an operation over meaning. A similarity encoder has none: the tokens for “elephant” are present, they pull the conditioning vector toward elephants, and the word “no” contributes very little in the other direction. This is the single most reliable difference between the two skills, and it generalises to “without”, “avoid”, and “not”.

The mechanism-appropriate fix is a negative prompt, which many systems expose as a separate field. It is not a phrasing trick; it is a second conditioning vector that the sampler steers away from, which is the operation you actually wanted. Where no negative prompt field exists, describe the positive state instead — “an empty savannah at dawn” rather than “a savannah with no elephants”.

The 77-token ceiling and what it did to style

CLIP’s text encoder has a context length of 77 tokens. That is not a soft preference — text past it is truncated, and in the older stacks it simply never influenced the image. This is the origin of the comma-salad style that still dominates prompt-sharing sites:

old style, dense and keyword-shaped, built for a 77-token encoder:
  portrait of an elderly fisherman, weathered face, golden hour,
  85mm, shallow depth of field, film grain, muted palette

newer style, for a model with a language-model text encoder:
  A close portrait of an elderly fisherman at golden hour. He is
  looking slightly off camera. Shot on 85mm with a shallow depth
  of field, so the harbour behind him is soft. Muted colours.

Both are legitimate; they target different machinery. Discovering which one your model wants takes about five minutes and saves a great deal of guessing — write both, fix the seed, and see which produced the image you asked for.

Rules that transfer and rules that do not

Iterating with seeds

This is the technique with no counterpart in text work and it is worth more than any phrasing advice on this page. Fix the seed and change one clause. Because the noise the sampler starts from is identical, the difference in the output is attributable to your edit rather than to chance. Change the seed and you are comparing two different images produced by two different prompts, which tells you nothing.

Keep a small file of prompt, seed, parameters and output path for every generation you liked. Reproducing an image six weeks later is otherwise impossible, and “make another one like that” is the most common request any image pipeline receives.

Which raises the technique that actually solves style consistency, and it is not a prompt technique at all. Wording will not reliably produce the same look twice, because the space of images matching any description is enormous. Reference images will: most systems accept an image as a style or subject reference, and the open-weight ecosystem has an entire layer of adapters — LoRAs and IP-Adapter-style conditioning — built to inject a visual identity that no amount of adjectives can specify. If your requirement is “every illustration on the site looks like it came from the same artist”, that requirement is met with a reference, a fixed seed and a fixed parameter set, not with a better sentence.

Two parameters round this out and are frequently mistaken for prompt problems. Aspect ratio is almost always an argument rather than words — asking for “a wide banner” in the prompt produces a composition that looks wide inside a square image, which is not what anyone wanted. And guidance scale, where exposed, controls how hard the sampler is pushed toward the prompt: too low and the model ignores you, too high and images become oversaturated and rigid. If your outputs have a burnt, over-contrasted quality, that is usually the guidance scale rather than the prompt.

One last asymmetry worth internalising: a hosted image API may rewrite your prompt before generating, expanding it into something longer and more detailed. When that happens your careful phrasing is an input to a rewriter rather than to the model, which explains the otherwise baffling experience of a short prompt returning something far more elaborate than you asked for. Where the API returns the revised prompt, log it — it is the only way to know what was actually generated from.