A clerk’s ordinance bot team wants a fair comparison of two rerankers before changing production RAG. What evaluation approach should they use?

Running an A/B framework that compares both rerankers on the same fixed municipal evaluation set is the fair approach before changing production RAG for a clerk’s ordinance bot. Shared queries, shared relevance labels, and a common scorecard isolate reranker effects from corpus drift or prompt changes. That controlled comparison is what GenAIOps needs to justify promoting one reranker over another for ordinance retrieval.

Scoring both rerankers on the same fixed municipal training set using precision at k fails because the training set rewards memorization and risks leakage, so it cannot prove generalization; a fair comparison requires a held-out evaluation set with graded relevance labels. Comparing both rerankers on the same fixed municipal evaluation set using BLEU scores is wrong because BLEU measures generated-text n-gram overlap, not retrieval ranking quality; rerankers need relevance metrics like precision at k or nDCG. Ranking both rerankers on the same fixed municipal evaluation set by average latency is wrong because latency measures speed, not relevance; even under identical conditions, a fast reranker can still return poor ordinance passages.

Exam caveat: A/B or offline pairwise comparison on a fixed eval set is about held-out conditions and relevance metrics—not training-set precision at k, BLEU scores, or latency rankings. Distinguish offline reranker bake-offs from production canary cutovers; both need metrics, but the stem asks for fair comparison before changing production.

Operational check: freeze an ordinance eval set with graded passages, score both rerankers with identical candidate pools and metrics (nDCG, precision@k, grounded citation rate), document the winner and effect size, then canary the winner only after the offline gate passes.