Vision Transformers

A vision transformer cuts an image into fixed squares, treats each square as a token, and runs a standard transformer encoder over them. The architecture is almost unchanged from text; what changes is that the model is given none of the assumptions a convolutional network has built in, and the original paper measured what that costs in training data.

An image becomes 196 tokens

input     224 x 224 x 3
patch     16 x 16
patches   (224/16) * (224/16) = 14 * 14 = 196
each patch flattened: 16 * 16 * 3 = 768 numbers

linear projection 768 -> D (768 for ViT-Base)
prepend a learnable [class] token           -> 197 tokens
add learned positional embeddings           -> 197 x 768

then: 12 transformer encoder layers, 12 heads each.

That is the whole of the vision-specific part. Everything after the third line is the same block used in a language model: multi-head attention, an MLP, residual connections and layer normalisation. The class token is a slot with no image content whose output vector is used as the summary of the image, which is a trick borrowed directly from BERT.

The attention is not causal. Every patch attends to every other patch in both directions from layer one, because an image has no reading order to preserve. This makes ViT an encoder, not a decoder, and it is why a vision transformer inside a multimodal model produces a set of embeddings that a language decoder then consumes.

The patch embedding is a convolution

Worth spelling out because it is usually described as if it were something new. “Cut into 16 by 16 patches with no overlap and apply the same linear projection to each” is exactly a convolution with kernel 16, stride 16 and 768 output channels. Most implementations write it that way, as a single Conv2d call.

So the one convolution in a vision transformer is the first operation, and after that there are none. The patch grid also fixes the model’s finest resolution: anything smaller than 16 pixels is mixed together in the projection and cannot be separated afterwards, which is why small-object detection was an early weakness and why smaller patch sizes exist at proportionally higher cost.

Resolution, priced

Attention cost grows with the square of the token count, and token count grows with the square of the side length. That compounds:

224 x 224, patch 16:  196 patches + 1 = 197 tokens
                      197^2 = 38,809 attention pairs per head per layer

384 x 384, patch 16:  576 patches + 1 = 577 tokens
                      577^2 = 332,929 pairs

pixels:          384^2 / 224^2 = 2.94x
attention pairs: 332,929 / 38,809 = 8.58x

Three times the pixels, eight and a half times the attention work. Go to 1024 by 1024 with the same patch size and it is 4,097 tokens and 16.8 million pairs — 432 times the cost of the 224 configuration for 21 times the pixels. This is the arithmetic behind every windowed, hierarchical or pooled vision transformer variant, and it is the arithmetic behind the price of a high-detail image in a multimodal API.

There is a second, less obvious cost of changing resolution: the positional embeddings are learned per position, so a model trained at 196 positions has 196 of them. Running at 576 positions requires interpolating that grid, which works but is an approximation, and it is why fine-tuning at the target resolution is standard practice.

What the patches become inside a multimodal model

Outside classification, the class token is usually ignored and the per-patch output vectors are what matter. In a vision-language model they are projected into the language model’s embedding space — often by a two-layer MLP, nothing more elaborate — and inserted into its input sequence as though they were tokens.

Which means an image consumes context, and the arithmetic is worth knowing before sending one:

336 x 336 image, patch 14:
  (336/14)^2 = 24 * 24 = 576 patch vectors
             = 576 tokens of context, for one image

High-detail tiling of a 1024 x 1024 image:
  4 tiles of 512, each resized to 336        -> 4 * 576 = 2,304
  plus one downsampled overview of the whole -> +   576
                                                  -------
                                                    2,880 tokens

Nearly three thousand tokens for one screenshot, which is more than most of the text around it. That is the mechanism behind image pricing and behind the fact that a conversation with a few screenshots in it fills a context window far faster than the word count suggests.

Two mitigations are common and both are visible trade-offs. Pooling adjacent patch vectors — merging each 2 by 2 group into one — cuts the token count by four at the cost of spatial precision. And resolution tiers, where the caller chooses a low-detail or high-detail mode, are the same decision exposed as an API parameter.

One further use worth naming: features from self-supervised vision transformers turn out to contain segmentation-quality object information without ever being trained on segmentation labels, which is why they are used as general-purpose visual features well outside the task they were trained on.

The inductive bias that was traded away

A convolution hard-codes two assumptions, as the CNN page sets out: locality, and translation equivariance. A vision transformer assumes neither. In layer one, any patch may attend to any other, and position is known only through a learned embedding that the model is free to use or ignore.

Removing an assumption raises the ceiling and raises the data requirement, and the original ViT paper (Dosovitskiy and colleagues, published 2020 and presented in 2021) reported both sides of that directly. Trained on ImageNet-1k alone, roughly 1.3 million images, ViT underperformed a comparable ResNet. Pre-trained on ImageNet-21k it drew level. Pre-trained on JFT-300M, around 300 million images, it overtook the convolutional baselines.

That is the trade priced in the only currency that matters here: the convolutional prior was worth roughly two orders of magnitude of labelled data. Below that threshold, hard-coding the assumption wins; above it, learning your own beats being told.

How the bias was bought back cheaply

The follow-up work is mostly about getting the ceiling without paying the data bill:

• Distillation and augmentation. DeiT (2021) trained a ViT on ImageNet-1k alone to competitive accuracy using heavy augmentation and a distillation token supervised by a convolutional teacher. A CNN taught the transformer the prior it did not have.
• Reintroduce locality. Swin restricts attention to local windows that shift between layers, and pools between stages, so the model regains a hierarchy and a cost that is linear rather than quadratic in image size — the same argument as sliding-window attention in text models.
• Self-supervision. Masked autoencoding — mask 75 per cent of the patches and reconstruct them — and contrastive self-distillation removed the need for hundreds of millions of labelled images, which is the part that was actually scarce.
• Better position handling. Relative and rotary position schemes generalise across resolutions better than a learned absolute grid.

Why it won anyway

Not on accuracy per FLOP at small scale, where a good CNN is still competitive or better. It won on three structural properties:

• One architecture for every modality. Once an image is a sequence of embeddings, it can be concatenated with text embeddings and fed to the same stack. Every current vision-language model works this way, and that is entirely a consequence of the tokenisation choice.
• Scale behaves. ViTs keep improving as parameters and data grow, in the way convolutional stacks did not.
• The whole toolchain transfers. The kernels, the parallelism strategies, the quantisation formats and the serving stacks all already existed for transformers.

The honest summary is that a vision transformer is not a better image model than a CNN at every size. It is a better component, because it emits the same currency — tokens — that everything else in a modern system consumes.