A RAG system for scientific literature uses a reranker to improve retrieval quality. The team is evaluating ColBERT (late interaction) versus a cross-encoder reranker. The corpus has 10 million chunks and the system must return results in under 200ms end-to-end. Which reranking approach is architecturally preferable, and why?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — b is correct because ColBERT's late-interaction architecture precomputes token-level embeddings for all corpus documents and stores them in an index (e.g., ColBERT's PLAID index). At query time, query tokens are encoded once and MaxSim scores are computed against the precomputed document representations — this is far faster than running a cross-encoder forward pass for each of the top-50 to top-200 candidates.
Full explanation below image
Full Explanation
B is correct because ColBERT's late-interaction architecture precomputes token-level embeddings for all corpus documents and stores them in an index (e.g., ColBERT's PLAID index). At query time, query tokens are encoded once and MaxSim scores are computed against the precomputed document representations — this is far faster than running a cross-encoder forward pass for each of the top-50 to top-200 candidates. For a 200ms end-to-end budget on a 10M-chunk corpus, cross-encoder reranking at scale would typically exceed this budget unless candidates are severely restricted. C is wrong because while cross-encoders produce accurate scores, their computational cost scales with the number of candidates; applying them to even 50 candidates in sequence is often too slow for sub-200ms requirements. A is wrong because the primary advantage of ColBERT over cross-encoders is not model size but the precomputed index pattern; parameter count comparison alone is not the correct rationale. B is wrong because cross-encoder rerankers are not typically cached per query — query-document pairs are unique, and caching at this granularity would require a prohibitively large cache.