Multimodal RAG: Retrieving Over Images and Text

Retrieval over a text corpus is a solved shape: embed chunks, embed the query, take the nearest neighbours. The moment the corpus contains diagrams, screenshots, scanned pages or photographs, you have to decide what a picture is indexed as — and the three available answers differ by orders of magnitude in storage and by entire categories in what they can find.

The problem in one sentence

A vector index compares things in one space. A query is text. If the corpus is pixels, either the pixels have to enter the text space, the text has to enter a shared space with pixels, or the query has to be answered by something other than nearest-neighbour search.

Three architectures

1 · Caption then index

Run every image through a vision model once at ingestion, store the generated description as text, and index it with your existing text pipeline. Nothing about your retrieval stack changes; you have simply added a preprocessing step.

It is the cheapest to operate and the easiest to debug — you can read the index, which is not a small advantage. Its ceiling is set at ingestion: whatever the captioner did not mention is unfindable forever. A caption saying “a bar chart of quarterly revenue” cannot answer “which quarter dipped below €2m”. The fix is to caption for your queries rather than generically — extract the underlying values into the caption if that is what people will ask about.

2 · Joint embedding

Use a model trained to put images and text in one shared space — the CLIP family, and its many successors. An image and its description land near each other, so a text query retrieves images directly with no caption in between.

This is the right architecture for photograph-shaped corpora: product catalogues, stock libraries, media archives. It is a poor one for documents, because a contrastive image/caption objective was never asked to represent the sentence in paragraph four of a page.

3 · Retrieve over page images directly

The newer approach, and for document corpora usually the strongest. ColPali (Faysse et al., 2024) skips text extraction entirely: it embeds each page image as a set of per-patch vectors using a vision-language model, embeds the query as per-token vectors, and scores with a late-interaction operator in the manner of ColBERT — every query token finds its best-matching patch and the scores are summed. Because the representation is per-patch, a match can be localised to a region of the page, and layout, figures and tables are represented natively because no flattening to text ever happened.

The storage arithmetic

This is the axis people miss until the bill arrives. The published ColPali configuration uses 128-dimensional vectors per image patch, with on the order of a thousand patches per page.

single dense vector per page
  1 x 1024 dims x 4 bytes (float32)        =   4.1 kB / page
  same at 2 bytes (float16)                =   2.0 kB / page

late interaction, per-patch
  1030 patches x 128 dims x 2 bytes        = 264 kB / page
                                             ~64x the dense index

100,000 pages
  dense, float16                           = 200 MB
  per-patch, float16                       =  26 GB

Twenty-six gigabytes is not prohibitive, but it is a different class of infrastructure decision from two hundred megabytes, and the query cost differs too: late interaction scores many vectors per candidate rather than one, so it usually runs as a reranking stage over a cheap first-pass retrieval rather than as the only index. Binary quantisation and pooling reduce the multiplier substantially and are standard practice; budget for the engineering, not just the disk.

How each one fails

Choosing, and the hybrid

• Photographs, products, media. Joint embedding. This is what it was built for.
• Scanned or visually complex documents. Page-image retrieval, with a cheaper first stage in front of it.
• Mostly text with a few diagrams. Caption-then-index. Do not rebuild a working pipeline for 4 % of the corpus; caption the diagrams into it.
• Exact identifiers matter. Keep a lexical index alongside whatever else you do. No embedding retrieves a part number as reliably as an inverted index does.

One implementation detail that applies to all of them: retrieval and generation do not have to consume the same representation. Retrieve over whatever works, then send the model the page image itself. The index exists to find the page; the model should see the real thing.

Granularity is the decision that follows, and it is where multimodal RAG differs most sharply from the text case. In a text pipeline you chunk to a few hundred tokens and retrieve several chunks cheaply. A page image is indivisible in practice — you cannot send half a scanned page and expect the layout to make sense — and it costs on the order of a thousand tokens. So retrieving the top ten pages, which would be unremarkable in a text system, is a five-figure token prompt before the question is even asked. Retrieve fewer, rerank harder, and consider a two-stage generation: a cheap pass over extracted text from twenty candidate pages to decide which three matter, then the expensive pass over just those three as images. The cost curve rewards precision here in a way it does not for text.