Visual Question Answering: How a Model Answers a Question About an Image

The image encoder and the text encoder produce vectors in two spaces that have nothing to do with each other. Everything interesting about visual question answering is in the operator that joins them, and that operator has been replaced four times.

What the model is actually asked to do

The task is a pair in, a string out: an image and a natural-language question, and an answer. What is easy to miss is that for most of the field’s history it was not a generation task at all. The benchmark answers are short — “yes”, “2”, “blue”, “tennis” — so the standard setup fixed a vocabulary of the few thousand most frequent answers in the training set and trained a classifier over it. A model that answers 3,000-way multiple choice is a very different object from one that writes a sentence, and comparisons across that boundary need care.

The other structural fact is that the question is not optional context. The same image supports “how many people are there”, “what colour is the umbrella” and “is it raining”, and the useful image representation is different for each. So the fusion cannot be “encode the image, encode the question, concatenate” if you want good answers — the question has to reach back and change what is read out of the image. Every architectural generation below is a different answer to that.

Four generations of fusion

Joint embedding. The original VQA baseline from Antol and colleagues in “VQA: Visual Question Answering” (2015) took a CNN’s global pooled feature for the image, an LSTM’s final hidden state for the question, projected both to a common width, and combined them by element-wise product before a softmax classifier. The element-wise product is worth noticing: it is a rank-one bilinear interaction, the cheapest possible way to let each image dimension modulate each question dimension. It works far better than concatenation and far worse than anything since, and it cannot attend — the image is one vector, so the question cannot look somewhere in particular.

Bilinear pooling. The obvious generalisation is the full outer product of the two vectors, which for typical widths is millions of dimensions and a projection matrix of billions of parameters. The literature is a sequence of approximations to it: Multimodal Compact Bilinear pooling approximates the outer product with count sketches and multiplication in the Fourier domain; Multimodal Low-rank Bilinear pooling and MUTAN replace it with low-rank and Tucker decompositions. These bought real gains and are worth knowing about because they name the actual problem — a rich interaction between two modalities is expensive, and everything since is a cheaper way to get one.

Attention over regions. The step that mattered most was giving the question somewhere to look. Anderson and colleagues, in the bottom-up top-down work of 2018, replaced the single pooled image vector with a set of region features taken from an object detector — on the order of a few dozen boxes per image, each with its own vector. The question vector then computes an attention weight per region, and the image representation is the weighted sum. That is the “top-down” half; “bottom-up” is the detector proposing salient regions in the first place. Suddenly “what colour is the umbrella” could read the umbrella region rather than the whole scene.

Cross-attention transformers. The next generation made the interaction two-way and deep. Models in the LXMERT and ViLBERT family run separate transformer streams over text tokens and region tokens, with co-attention layers where each stream attends to the other, repeatedly, at several depths. They are pretrained on image-caption corpora with masked-token and image-text-matching objectives and then fine-tuned for VQA. Fusion is no longer a single operator at the end; it is distributed through the network.

When fusion became ordinary self-attention

The current arrangement is simpler than any of the above, and the simplification is the point. A frozen vision transformer — usually a CLIP-pretrained one — encodes the image into a grid of patch embeddings. A projector maps those embeddings into the language model’s token embedding space. Then they are concatenated with the question’s text tokens and the language model runs normally. There is no fusion module. The image is literally in the prompt, and fusion happens in the self-attention the model was already doing.

The two well-known projectors differ mainly in how much they compress. BLIP-2 introduced a Q-Former: a small transformer holding a fixed set of learned query tokens that cross-attend to the image patches and emit a fixed, short sequence regardless of image size. LLaVA-style models use a linear or two-layer MLP projection and pass through one token per patch. Compression matters because those tokens are billed and attended like any others — a high-resolution image tiled into several crops can cost more context than the question by two orders of magnitude, which is why hosted APIs expose resolution or detail settings at all. See how a vision transformer turns an image into tokens and what a detail setting changes.

How an answer is scored

The VQA benchmark collects ten independent human answers per question and scores a candidate answer as

accuracy = min( (number of humans who gave this exact answer) / 3 , 1 )

and then averages that over all ten choose nine subsets of the annotators, so a model is never credited for matching an annotator it is being compared against. Three humans agreeing with you is full marks; one is a third. Answers are normalised first — lowercased, articles stripped, contractions and number words standardised — which handles “2” against “two” but not “a couple”, and not “there are two people in the image”. A fluent generative model scores badly on this metric for reasons that have nothing to do with being wrong, and that is the main thing to know before reading a VQA leaderboard.

The other thing to know is why the benchmark was rebuilt. The first version had a strong language prior: enough questions had the same answer regardless of image that a model ignoring the image entirely scored respectably, and “is there a...” could be answered “yes” most of the time. Goyal and colleagues rebalanced it in “Making the V in VQA Matter” (2017) by pairing each question with two similar images that have different answers. A model that ignores the image now scores near chance by construction. Any VQA number quoted without saying which version it is on is not comparable.

What fusion does not fix

• Counting. Attention pools; it does not enumerate. A weighted sum over regions loses the distinction between one strongly attended object and three weakly attended ones, and models consistently answer small counts by prior. See why counting needs a different formulation.
• Reading text in the image. Patch embeddings at typical resolutions do not resolve small print, and the answer vocabulary cannot contain arbitrary strings. Questions about signage, labels and receipts need an OCR path feeding the model, which is a pipeline decision rather than an architectural one.
• Spatial relations and negation. “What is to the left of the lamp” and “which shelf has no price tag” both require composing over the scene rather than attending to part of it. Attention weights are not a relational representation, and nothing in the training objective forces one.
• Unanswerable questions. Asked about something not in the image, a model trained to always emit an answer emits one. The benchmark fix is to include unanswerable examples; the deployment fix is to treat a confident answer about an absent object as the expected failure rather than a surprise. See how vision models hallucinate objects.

A useful boundary: a model with a text-conditioned open-vocabulary detector in front of it answers “is there a forklift” far more reliably than a VQA model does, because the question is being answered by a component whose output is a box and a score rather than a token. Where the question decomposes into detection, detect.