Sparse, Dense and Late-Interaction Retrieval (ColBERT)

Single-vector dense retrieval compresses a document into one point before it has any idea what will be asked of it. Cross-encoders refuse to compress at all and pay for it on every query. Late interaction is the design that sits between them, and its cost is entirely in storage.

Three paradigms on one axis

The axis is when the query meets the document. Everything else follows from where a system sits on it.

“Late” in late interaction is relative to a cross-encoder, where interaction is early — inside the model. ColBERT defers it to after both sides have been independently encoded, which is precisely what makes the document side precomputable and therefore indexable.

What late interaction computes

ColBERT (Khattab and Zaharia, SIGIR 2020) encodes a document into one low-dimensional vector per token and scores with MaxSim:

score(q, d) = sum over query tokens i of
                  max over document tokens j of  cosine(q_i, d_j)

query "refund window after delivery", document = a returns policy

  "refund"    -> best match on document token "refunded"   0.91
  "window"    -> best match on "within 30 days"            0.74
  "after"     -> best match on "following"                 0.62
  "delivery"  -> best match on "delivered"                 0.88
                                                    total  3.15

Each query token independently finds its best evidence anywhere in the document, and the score is the sum. This is why late interaction handles multi-aspect queries that single-vector retrieval loses: a document that answers three of your four requirements strongly and one weakly scores accordingly, whereas one pooled vector has already averaged all four together into a point that may be near neither.

It also explains the resilience to rare terms. A single-vector model dilutes an unusual token among hundreds of others; MaxSim lets that one token contribute its full match score. Late interaction sits closer to lexical matching in behaviour than dense retrieval does, while still matching on meaning rather than on string equality.

What the papers actually report

No benchmark was run for this page, so what follows is what the authors published. The original ColBERT paper reports retrieval effectiveness competitive with BERT-based reranking pipelines on MS MARCO passage ranking while being orders of magnitude cheaper per query than running a cross-encoder over the candidates, which was the central claim: most of the quality of late-stage interaction, at retrieval latency.

ColBERTv2 (Santhanam et al., NAACL 2022) is the practical successor. Its contribution is residual compression — each token vector is stored as the id of a nearby centroid plus a heavily quantised residual — and the paper reports a 6 to 10× reduction in index size relative to ColBERT while improving quality, evaluated across BEIR and other out-of-domain sets. The companion PLAID work addresses the retrieval engine itself, pruning candidate documents by centroid before any full scoring happens.

Take from that what the authors claim and no more: late interaction is a real quality gain over single-vector dense retrieval, particularly out of domain, and the entire engineering effort in the line of work has gone into making its index affordable. That second fact is the honest headline.

The storage arithmetic, which is the catch

One vector per token instead of one per chunk. For 100 million chunks averaging 400 tokens — 40 billion token vectors — at ColBERT’s 128-dimensional token embeddings:

single-vector dense, 1536 dims float32:
    100e6 * 1536 * 4                        =    614 GB

ColBERT, 128 dims float16, one per token:
    40e9  * 128  * 2                        = 10,240 GB   (16.7x)

ColBERTv2 residual compression, taking the paper's 6-10x reduction
(so roughly 26-43 bytes per token vector):
    40e9  * 32                              =  1,280 GB   (2.1x)

Even with the compression the papers describe, the index is around twice a full-precision single-vector one — and against a binary- quantised dense index at 19 GB, it is nearly two orders of magnitude larger. That comparison, not any quality number, is what decides whether late interaction is available to you.

Query cost has the same shape. A 32-token query against a 400-token document is 12,800 dot products in 128 dimensions for a single document, against one dot product in 1536 dimensions for the dense case. PLAID-style centroid pruning is what keeps this tractable, and it is why you should use a mature implementation rather than writing MaxSim over a candidate list yourself and concluding the method is slow.

When to reach for it, and what else to try first

• Try hybrid retrieval first. BM25 fused with a dense retriever by reciprocal rank fusion captures a large share of what late interaction offers — resilience to rare terms and exact matches — at a fraction of the storage. If you have not built that, build it before you evaluate ColBERT.
• Then try a cross-encoder reranker. Retrieve 100 cheaply, rerank with a cross-encoder, return 10. This is the highest quality per unit of engineering effort available in retrieval today, it adds no storage at all, and it costs one model call per query on a bounded candidate set.
• Reach for late interaction when reranking latency is the binding constraint rather than quality — you want reranker-grade behaviour at first-stage speed — or when your queries are genuinely multi-aspect and long, which is where MaxSim’s per-token evidence is doing something the alternatives cannot.
• Rule it out on corpus size. Run the arithmetic above with your token count before anything else. If the compressed index does not fit your budget, no quality result changes the answer.

The broader point is that these three paradigms are not a progression and the newest is not the default. They differ in what they precompute, and what you can afford to precompute is a property of your corpus, not of the state of the art.