A city RAG agent’s evaluation should distinguish bad retrieval from bad generation. What pipeline design achieves that?
Select an answer to reveal the explanation.
Short Explanation
Wrong pamphlet versus wrong summary of the right pamphlet are different bugs. Score retrieval hit-rate and answer faithfulness separately so you know which to fix. One vague thumbs-up hides the diagnosis.
Full Explanation
A city RAG agent’s evaluation must separate retrieval failures from generation failures. Measuring retrieval hit-rate separately from final-answer faithfulness to city documents achieves that pipeline design: hit-rate shows whether the right municipal pamphlets, ordinances, or FAQs were retrieved; faithfulness shows whether the answer stayed true to those documents. Different bugs need different fixes—index/query tuning versus grounded generation controls—and collapsed scores hide the diagnosis.
Scoring only final prose style while ignoring whether city documents were retrieved rewards fluent invention. Collapsing retrieval and answer quality into one unexplained thumbs-up prevents root-cause analysis. Disabling retrieval metrics because generation quality always implies correct documents is false: polished prose can still cite the wrong pamphlet or none at all.
Exam caveat: define hit-rate against a labeled relevant-document set for each civic question, and define faithfulness against those retrieved (or gold) municipal passages. Operational check: on a permit or benefits RAG suite, report retrieval hit-rate and answer faithfulness as separate metrics, sample failures from each bucket, and assign retrieval owners versus generation/prompt owners accordingly before the next tuning cycle.