A transit disruption bot team wants evidence that a new retrieval strategy beats the current one. What evaluation approach fits?

A/B testing retrieval strategies and scoring answer relevance on a fixed eval set is correct because RAG optimization needs comparative evidence, not measuring chunk recall at k on labeled passages for the new strategy alone, which lacks a baseline comparison and uses a retrieval proxy rather than proof of better disruption answers. A transit disruption bot team that wants proof a new retrieval strategy beats the current one should run controlled A/B (or offline pairwise) comparisons against the same municipal eval questions and score answer relevance—and ideally retrieval hit quality—so promotion is data-driven. Fixed eval sets keep the comparison fair across strategies and time.

A/B testing retrieval strategies and scoring answer relevance on separate samples runs a comparison, but using different samples confounds the result because differences in query difficulty can masquerade as strategy gains instead of proving more relevant disruption answers on the same questions. Measuring chunk recall at k on labeled passages for the new strategy alone tracks a retrieval proxy without a baseline, so it cannot show whether the new strategy produces more relevant disruption answers than the current one for riders. Comparing both retrieval strategies by scoring answer relevance on training passages tests on data the system may have already seen, so it risks overfitting and does not prove better rider-facing answer quality on new disruption questions.

Exam caveat: retrieval strategy changes are validated with relevance metrics on a fixed eval set and/or structured A/B tests—not chunk recall for the new strategy alone, not answer relevance on training passages, not A/B comparisons on separate samples.

Operational check: freeze a transit-disruption eval set with expected answers or relevant document IDs, run baseline and candidate retrieval strategies, score relevance (and groundedness), optionally canary A/B in production with guardrails, promote the winner when gains clear the team’s threshold, and archive the comparison report for the AI CoE.