How a Multimodal Model Sees an Image

There is no separate “vision part” of the model doing the looking. An image is converted into a run of vectors that sit in the context next to your words, and from the transformer’s point of view they are just more tokens. Everything strange about how these models behave with pictures is a consequence of that conversion.

Step one: the image becomes patches

The move that made this work is from Dosovitskiy et al.’s 2020 vision transformer paper, whose title is the whole idea: “An Image Is Worth 16x16 Words”. Cut the image into a grid of fixed-size squares, flatten each square, run it through one linear layer, and you have a sequence of vectors — exactly the shape a transformer eats.

The arithmetic is worth doing once by hand because it is the reason every later number in this cluster is what it is. Take a CLIP ViT-L/14 encoder at 336 × 336, the one LLaVA-1.5 used:

image     336 x 336 px
patch      14 x 14 px
grid      336/14 = 24 across, 24 down
patches   24 * 24 = 576 vectors

double the image side to 672 x 672:
grid      48 x 48
patches   2304 vectors   <- 4x, not 2x

Patch count grows with the area. That single line explains why image pricing is quadratic in the long edge, why providers cap resolution, and why a screenshot of a 4K monitor is an expensive thing to send.

Step two: an encoder, trained separately

Those patch vectors do not go straight into the language model. They go through a vision encoder that was trained on its own objective, most often a CLIP-style contrastive one — Radford et al., 2021 — where an image tower and a text tower are pulled together for matching pairs and pushed apart for mismatched ones, over a very large scrape of image/caption pairs from the web.

That training objective is a fact about your production system, not trivia. A contrastive encoder is rewarded for producing a representation that matches a caption. Captions say “a dog on a beach”. They do not say “the third value in the second column is 41.2”. The encoder is therefore excellent at gist and structurally indifferent to small, dense, spatially precise detail — which is where nearly every complaint about vision models originates.

Step three: the projector

The encoder’s output lives in its own vector space with its own dimensionality. The language model expects its own. Between them sits a small trained module, and there are two dominant designs:

• Projection. A linear layer or two-layer MLP maps each image vector into the language model’s embedding space, and the results are inserted into the sequence as if they were tokens. This is the LLaVA design (Liu et al., 2023) and its appeal is that it is almost nothing: the language model is unchanged, and the image occupies real positions in the context window.
• Cross-attention. The image vectors stay outside the sequence and new attention layers inside the language model read from them. This is the Flamingo design (Alayrac et al., 2022), which also introduced a resampler that compresses a variable number of patches down to a fixed small number of latent vectors. It keeps the text context free but requires surgery on the language model.

If a provider bills you for image tokens against your context window, you are almost certainly looking at the first design. If images seem not to consume context at all, the second is a good guess.

In the projection design the mechanics of insertion are worth knowing, because they explain the ordering effects people notice. The chat template contains a placeholder token for the image, and at inference the runtime expands that single position into however many vectors the encoder produced. From that point the image occupies a contiguous block of positions in the sequence and is subject to exactly the same causal attention as text: tokens after it can attend to it, tokens before it cannot. That is the entire reason the order of your content matters. A question placed before the image is answered by a model that has not yet seen the image at the positions where it is thinking about the question; a question placed after it is not. Put the image first and the instruction last unless you have a specific reason to do otherwise.

Why resolution is the whole story

A fixed-grid encoder has one input size. Feed it a 3000 ×  2000 scan of a page and the image is downscaled to fit, so the 8-point footnote that occupied 11 pixels of height now occupies two. The information is gone before the transformer is involved. No prompt recovers it.

The fix that current model families converged on is tiling: split the large image into several encoder-sized crops, encode each, and also encode a downscaled version of the whole thing so the model retains global layout. Variants of this appear across vendors, and NaViT (Dehghani et al., 2023) pushed further by packing variable-resolution images into one sequence rather than forcing a square. Tiling is why the published pricing formulas count tiles, and why a tall thin image and a square image with the same pixel count can cost different amounts.

What the pipeline predicts about failures

Each row is the same diagnosis in a different costume: the model can only reason about what survived the conversion to tokens. Before rewriting a prompt, ask what the encoder actually received.