A RAG system for scientific literature uses a reranker to improve retrieval quality. The team is evaluating ColBERT (late interaction) versus a cross-encoder reranker. The corpus has 10 million chunks and the system must return results in under 200ms end-to-end. Which reranking approach is architecturally preferable, and why?

The correct approach is ColBERT late interaction, because it precomputes token-level embeddings for all corpus documents and stores them in an index, such as ColBERT's PLAID index. At query time, query tokens are encoded once and MaxSim scores are computed against the precomputed document representations; this is far faster than running a cross-encoder forward pass for each of the top-50 to top-200 candidates. For a 200ms end-to-end budget on a 10M-chunk corpus, cross-encoder reranking at scale would typically exceed this budget unless candidates are severely restricted. The cross-encoder approach that relies on full attention between query and each candidate to produce higher relevance scores at any latency budget is wrong because, while cross-encoders produce accurate scores, their computational cost scales with the number of candidates; applying them to even 50 candidates in sequence is often too slow for sub-200ms requirements. The ColBERT approach justified by smaller model size or fewer parameters reducing GPU memory and infrastructure cost is wrong because the primary advantage of ColBERT over cross-encoders is not model size but the precomputed index pattern; parameter count comparison alone is not the correct rationale. The cross-encoder approach that claims scores can be cached per query and reused for repeated queries against the same top-50 candidates is wrong because cross-encoder rerankers are not typically cached per query; query-document pairs are unique, and caching at this granularity would require a prohibitively large cache.