A late-interaction neural retrieval model that encodes queries and documents into per-token embedding matrices with BERT and scores relevance by summing each query token’s maximum similarity over document tokens, retaining much of the accuracy of full cross-encoder attention whilst permitting document embeddings to be precomputed and indexed for scalable, fast retrieval.
Semantic Classification
Content
Definition
ColBERT (Contextualised Late Interaction over BERT), introduced by Khattab and Zaharia at SIGIR 2020, occupies the middle ground between two retrieval paradigms. Single-vector Dense Retrieval models such as Dense Passage Retrieval compress an entire passage into one embedding, which is cheap to index and search but loses fine-grained term-level evidence. Cross-encoders concatenate query and document and run full BERT attention over the pair, which is highly accurate but far too slow to score more than a shortlist — hence their use in Cross-Encoder Reranking. ColBERT’s late interaction keeps one embedding per token on both sides, deferring query–document interaction to a lightweight final step.
Scoring uses the MaxSim operator: for each query token embedding, take the maximum cosine (or dot-product) similarity across all document token embeddings, then sum these maxima over query tokens. Because documents never attend to the query during encoding, all document token matrices can be precomputed offline and indexed; at query time only the short query is encoded, and candidate retrieval proceeds through an approximate nearest-neighbour index over token vectors followed by exact MaxSim scoring. The result recovers most of the effectiveness gap to cross-encoders whilst remaining orders of magnitude faster.
ColBERTv2 (Santhanam et al., 2022) made the approach practical at scale, combining residual compression — quantising token embeddings against learned centroids to cut storage roughly 6–10× — with denoised supervision distilled from a cross-encoder. The accompanying PLAID engine accelerated retrieval further through centroid-based candidate pruning. On the BEIR Benchmark, ColBERTv2 became a standard strong baseline for zero-shot retrieval, out-of-domain robustness being a recognised strength of token-level matching over single-vector models.
Current Landscape
-
Native multi-vector support is now mainstream: Qdrant has offered first-class multivector collections with a MaxSim comparator since v1.10 (2024); Vespa supports tensor-field MaxSim ranking natively; Weaviate (v1.27+) and LanceDB followed, and Elasticsearch (8.18+) and OpenSearch (3.3+) added multi-vector support to their cores during 2025 — relevant wherever Semantic Search or Hybrid Search stacks are assembled.
-
Multilingual and long-context variants: Jina-ColBERT-v2 (August 2024, arXiv:2408.16672) extended late interaction to 89 languages and 8,192-token documents with Matryoshka output dimensions of 128/96/64, beating original ColBERTv2 by ~6.5% on 14 BEIR tasks; answerai-colbert-small showed strong quality at just 33M parameters.
-
ColPali and visual late interaction: ColPali (“Contextualised Late Interaction over PaliGemma”, arXiv:2407.01449, accepted at ICLR 2025) applies MaxSim scoring to vision-language-model patch embeddings of rendered pages (~1,000 patch vectors per page), enabling OCR-free document retrieval; Vespa has published work scaling ColPali-style retrieval to billions of pages with binary quantisation and phased ranking, and ColQwen variants extend the family.
-
Deployment pattern: in production, late interaction is predominantly used to rerank a small candidate set retrieved by a cheaper representation (single-vector or BM25) rather than as brute-force first-stage retrieval — Qdrant’s Query API expresses this retrieve-then-rerank flow in a single call.
-
Trade-offs: per-token storage (roughly 30× a single-vector index before quantisation, ~1,000× for page-image patches) remains the chief cost, motivating continued work on pooling, int8/binary quantisation, and token pruning.
Sources:
-
https://qdrant.tech/course/multi-vector-search/module-1/late-interaction-basics/