CLIP and Contrastive Multimodal Training

CLIP trains an image encoder and a text encoder to place a picture and its caption at the same point in one shared space. That single objective is what made zero-shot image classification, text-to-image conditioning and multimodal retrieval all fall out of the same set of weights.

Two encoders, one space

image_encoder : image -> R^d      a ViT or a ResNet
text_encoder  : text  -> R^d      a transformer, [EOS] token pooled

both outputs L2-normalised, so every vector lies on the unit sphere
and the dot product of two of them IS their cosine similarity.

d is typically 512 or 768.

Two separate towers with no cross-attention between them. That is a deliberate choice and it is what makes the model usable: an image can be encoded once and stored, a query encoded once and compared against millions of stored vectors with a dot product. A model that had to run a joint forward pass on every image-text pair would be more accurate per comparison and useless as a retrieval index.

The normalisation is not cosmetic. It removes vector magnitude from the comparison, which is exactly what you want when the two towers are different architectures trained on different modalities and have no reason to agree on scale. It is the same reason cosine similarity rather than raw dot product is the default in any retrieval system.

The loss is one matrix and its diagonal

Take a batch of N image-caption pairs. Encode all of both sides, and form every similarity:

I : (N, d)   normalised image vectors
T : (N, d)   normalised text vectors

S = (I @ T.T) / temperature      ->  (N, N)

S[i][j] = similarity of image i with caption j

labels = [0, 1, 2, ..., N-1]     the diagonal is the correct pairing

loss = 0.5 * cross_entropy(S, labels)        each row: find its caption
     + 0.5 * cross_entropy(S.T, labels)      each column: find its image

That is the whole training objective. Each row of the matrix is an N-way classification problem whose answer is the diagonal entry, and the loss is applied symmetrically so the space is shaped from both directions.

Notice what is being asked. Not “reconstruct the caption”, not “predict the image”, but “pick the right one out of this batch”. Everything about CLIP’s strengths and its weaknesses comes from that being the question.

Why batch size is a capability lever

The negatives are the other items in the batch. There is no separate negative sampling step; the batch is the negative set. So batch size directly controls how hard the task is:

batch 256:     each row has 1 positive and    255 negatives
batch 32,768:  each row has 1 positive and 32,767 negatives
               the similarity matrix is 32,768^2 = 1.07 billion entries

With 255 distractors, telling a dog photo from 255 random photos requires almost no detail — matching a couple of salient nouns is enough. With 32,767 distractors, several of them are also dogs, and the model has to encode breed, pose and setting to win. The gradient signal gets qualitatively richer as the batch grows, which is why contrastive training is one of the few places where batch size behaves like a capability parameter rather than a throughput one, and why these models are trained across many accelerators with the similarity matrix sharded.

The original CLIP (Radford and colleagues at OpenAI, 2021) reported a batch size of 32,768 and training on 400 million image-text pairs gathered from the web. Both numbers are load-bearing: the data made the coverage, the batch made the discrimination.

SigLIP: removing the global softmax

The batch-size argument has an ugly consequence in the training system. A softmax normalises across a whole row, so computing the loss requires every embedding in the batch to be available at once. Split the batch across many accelerators and each one must gather all the others’ vectors before it can compute anything, and somebody has to hold an N by N matrix.

Softmax loss:   needs the full row -> all-gather across devices
                             -> materialise N x N

Sigmoid loss:   each pair judged independently
                  positive on the diagonal, negative off it
                  loss = sum over pairs of a binary term
                  -> computable in chunks, no global normalisation

SigLIP (Zhai and colleagues at Google, 2023) makes exactly that substitution: replace the row-wise softmax with an independent sigmoid on every pair. The N by N matrix becomes N^2 separate binary decisions — is this image and this caption a match — with a learned bias term to cope with the extreme imbalance, since only N of the N^2 pairs are positive.

Because the loss decomposes over pairs, it can be computed a block at a time: each device handles its own chunk, passes embeddings around in a ring, and the full matrix never exists. The reported practical consequences are that very large batches get cheaper and, more usefully for most people, that small batches work better than the softmax version, because the loss no longer depends on having a large pool of in-batch negatives to normalise against.

It is a good example of the pattern that runs through this cluster: the change is not to what is learned but to the shape of the computation, and the shape is what decides whether something is trainable at scale.

What the temperature does

The similarities are divided by a temperature before the softmax, and in CLIP that temperature is learned rather than fixed — stored as a log-scale parameter and clipped to stop it running away.

Its effect is on how the loss allocates attention across negatives. All the cosines live in a narrow band near the top of the range, since normalised vectors of related things are all fairly close. Dividing by a small temperature stretches those small differences into large logit gaps, so the softmax concentrates on the few hardest negatives. A large temperature flattens everything and the loss spreads itself thinly over all of them, including the easy ones that teach nothing.

Left to itself, the parameter falls to a small value early in training, which is the model choosing to focus on hard negatives as soon as the easy ones are solved. It is a clean example of a hyperparameter that people used to tune becoming something the optimiser handles.

Zero-shot classification is retrieval

CLIP has no classification head and never saw a label set. Zero-shot classification works by turning the class names into text and doing a nearest-neighbour lookup:

1. Write each class as a sentence — a photo of a {label} — and encode all of them once. For 1,000 classes that is 1,000 vectors, computed once and cached.
2. Encode the image once.
3. Take the dot product against all 1,000 class vectors and return the argmax. Softmax the scores if you want probabilities.

Changing the label set is now editing a list of strings, not retraining. That is the property that made CLIP infrastructure rather than a model.

The phrasing matters more than it should, which is a hint about the limits below. “A photo of a {label}” outperforms the bare label because captions on the web look like sentences, not like nouns. The original work went further and ensembled predictions across dozens of templates — the same class described many ways, averaged in embedding space.

What the objective never learns

Go back to the question the loss asks: pick the right caption out of this batch. If the batch contains one picture of a dog and one caption mentioning a dog, matching on the word “dog” is sufficient. Nothing forces the model to encode which object is which colour, what is on top of what, or how many there are, because those details are almost never needed to win the batch.

• Attribute binding. “A red cube on a blue sphere” and “a blue cube on a red sphere” embed close together. The bag of concepts is the same and the objective never had to distinguish them.
• Counting. Three cats and five cats look alike to the text tower, because captions containing numbers are rare and the number is rarely the discriminating feature.
• Word order and negation. Contrastive text towers behave substantially like bag-of-words models, and “not a dog” is a well-known failure.
• Fine-grained and rare categories. Zero-shot accuracy tracks how often the concept appeared in web captions, which is a long-tailed distribution.
• Web bias, inherited whole. The training set is uncurated internet pairs, with the demographic and cultural skews that implies, and the model carries them into every downstream system that uses it as a filter or a scorer.

The trade: one loss bought a shared space where a dot product compares an image to a sentence, cheaply and at index scale, and generalises to categories nobody trained on. It paid with compositionality — the model learns what is present, not how the parts relate — and with a hard dependence on batch size and data volume that puts training one out of reach of most teams. Both halves matter when CLIP is the text encoder conditioning an image generator, which is where its weakness at relations shows up as pictures that contain the right objects doing the wrong thing.