A housing RAG team sees fluent answers that still miss the right policy sections. What evaluation design best isolates the failure?

Measuring retrieval relevance separately from final-answer fluency so retrieval bugs are not hidden by polished wording is the evaluation design that best isolates why a housing RAG team sees fluent answers that still miss the right policy sections. Fluency and retrieval are different failure modes: a generator can write smooth prose from the wrong chunks. Scoring retrieval relevance on its own exposes missing or off-policy sections before the final paragraph masks the defect. Scoring only how eloquent the final paragraph sounds and ignoring whether the right chunks were retrieved fails by concept: eloquence without correct retrieval produces confident wrong housing guidance. Deleting the retriever and evaluating only random Wikipedia paraphrases fails by concept—it abandons the housing corpus under test and measures an unrelated task. Treating token count as the sole measure of housing-policy correctness fails by concept: length does not equal correct policy coverage. Exam caveat: RAG evaluation must disentangle retrieve versus generate; fluency-only scoring is a common distractor. Operational check: build labeled queries with expected housing policy chunks, score retrieval recall/precision or nDCG separately from answer rubrics, alert when fluency stays high while retrieval relevance drops, and fix chunking or embeddings before rewriting generator prompts alone.