Why Image Models Struggle With Text and Hands

Text and hands fail for different reasons, and one of the reasons for text is not a training problem at all — it is a resolution budget. At a downsample factor of 8, a letter with an 8-pixel cap height occupies one latent cell. There is no representation in which that is a shape.

The short answer

There are four distinct causes and they are usually collapsed into one. Two are about text specifically, one is about hands specifically, and one applies to both.

• The autoencoder discards small glyphs. Arithmetic, not learning. Derived below.
• The text encoder does not represent characters as shapes. The prompt is tokenised into subwords, and the embedding for a word carries no reliable information about the sequence of letters it contains.
• The training objective has no discrete constraint. Nothing scores a word as right or wrong. Plausible ink in a plausible arrangement is what the loss rewards.
• Structure is committed before detail. The early steps allocate a region to “a hand” or “some writing”, and the late steps fill it convincingly without revisiting whether the allocation was right.

The arithmetic: a glyph does not fit

A latent diffusion model with downsample factor f = 8 works on a grid where each cell corresponds to an 8×8 pixel block. Take a 1024×1024 output and put some text in it.

Latent grid at f = 8:   128 × 128 cells,  one cell per 8 × 8 pixels

Text at various sizes in a 1024-pixel-tall image:

  cap height   pixels tall   latent cells tall   what is representable
  ------------------------------------------------------------------------
     small        12 px           1.5 cells      nothing; a smear
     caption      24 px           3.0 cells      a stroke pattern, not letters
     heading      64 px           8.0 cells      recognisable letterforms
     poster      160 px          20.0 cells      clean type

A letter needs enough cells to encode its distinguishing features: the
counter of an 'a', the crossbar of an 'e', the difference between 'rn'
and 'm'. Below roughly 4-6 cells of cap height there is no room for them.

This is not the diffusion model’s limitation. Test it directly: take a photograph containing small print, encode it with the autoencoder, decode it immediately, and look at the print. It is already unreadable, with no diffusion involved. The round-trip test is in the latent diffusion page, and it takes about a minute to run.

Two things follow. Increasing the latent channel count — 4 to 16 — raises the information budget per cell and improves this, because more can be encoded in the same spatial grid. And generating text large gets you text; asking for a paragraph on a book spine cannot work at any model quality.

The text encoder never saw the letters

The prompt goes through a text encoder that produces embedding vectors. Two properties of that pipeline hurt.

Tokenisation. Words are split into subword tokens, and a token is an atomic unit with a learned embedding. The model has no access to the characters inside it. This is the same mechanism that makes language models unable to count letters, explained in why AI cannot count the letters in a word, and it applies identically here.

The encoder’s own training objective. Contrastive image-text encoders are trained to align a whole caption with a whole image. That objective rewards representing what is in the picture and gives no pressure at all to preserve orthography — two captions differing by a letter usually describe the same image, so the embedding is free to treat them as near-identical.

A text encoder that is a full language model, rather than a contrastive image-text encoder, carries substantially more character-level structure, because its own training objective required it. That is one of the architectural changes discussed below.

Nothing in the objective rewards spelling

The training loss is squared error on a noise prediction. Consider two candidate outputs for a sign that should read OPEN: one reading OPEN and one reading OPFN. In pixel or latent space these differ by a small amount of ink in one small region. The loss difference is negligible.

Correct spelling is a discrete, global, all-or-nothing property, and the model is optimising a continuous, local, averaged one. Nothing in the architecture represents “this region is a word and words have exactly one correct rendering”. The model generates a plausible distribution of ink conditioned on a text-shaped region, which is why the output so often looks like writing in a language you almost recognise.

Hands are a different problem

Hands are not a resolution problem in the same way — a hand in a portrait usually has plenty of latent cells. Four things compound instead.

• Extremely high articulation. A hand has more than twenty independent joints. The set of physically valid configurations is a low-dimensional manifold inside a high-dimensional space, and the model has learned it only statistically.
• Enormous appearance variance in training data. Hands appear at every angle, partially occluded, cropped, holding things, overlapping each other. The conditional distribution at those pixels is very high entropy, and a sample from a high-entropy distribution is a plausible draw rather than the modal answer.
• Counting is a global discrete constraint. Exactly five fingers is the same kind of requirement as correct spelling, and the model has the same absence of machinery for it. Local plausibility is what the loss rewards, and a sixth finger is locally plausible everywhere along its length.
• Structure is committed early. If the early steps allocated an ambiguous blob at the end of an arm, the later steps resolve it into fingers with no mechanism to reconsider how many there should be. See why early steps decide composition.

The practical implications differ from text accordingly. Generating a hand larger does help, because more cells means the late steps have more room, but it does not fix counting. Inpainting the hand at crop resolution is the reliable production fix, and a pose or depth signal that specifies the finger positions removes the ambiguity at the source.

What architectural changes relax

Four classes of change address these causes, and it is worth being precise about which cause each one touches, because they are often reported as a single improvement.

A spelling test you can run

The protocol matters more than the score, because a score without a protocol is not comparable to anything.

1. Fix the target strings. Twenty of them, five to twelve characters, no repeated words. Include at least five that are not English words, since a real word can be produced from the model’s prior rather than from the conditioning.
2. Fix the prompt template. One template, with the string substituted, so nothing about phrasing varies between trials.
3. Sweep the rendered size, not just the prompt. Ask for the same string as a poster headline, a shop sign and a book-spine caption. This separates the resolution cause from the representation cause, and it is the step everybody skips.
4. Fix seeds and generate several per cell. Five seeds per string per size. Report the fraction of exact matches, and count an image as a match only if every character is correct.
5. Record everything. Model identifier and file hash, sampler, schedule, steps, guidance scale, resolution, date. Without these the result is an anecdote.
6. Run the autoencoder round-trip separately. Encode and decode a photograph of the same text at the same rendered sizes. That gives you the floor: no generation can beat it, and it tells you whether a failure is the compressor or the model.