A historic viaduct helper retrieves a 1910 paint recipe when asked about this Saturday’s detour. The final sentence is fluent; the retrieved passage was irrelevant. What should be scored separately?
Select an answer to reveal the explanation.
Short Explanation
Think of asking about Saturday's detour and getting a 1910 paint recipe. Score retrieval relevance, not only how pretty the last sentence is. A fluent line can sit on the wrong page.
Full Explanation
When a historic viaduct helper retrieves a 1910 paint recipe for Saturday’s detour question, the failure is retrieval relevance even if the final sentence is fluent. RAG systems must be scored on whether the retrieved passage actually answers the traveler’s query, not only on whether the generator writes smoothly. Fluent prose grounded in the wrong page is still a bad answer for municipal travelers who need current closure and detour guidance.
Final fluency alone cannot prove the passage was relevant to this weekend’s route. ROUGE against historical paint text would reward the wrong retrieval rather than measuring whether the chunk helps with detours. Claiming RAG applications never evaluate the retrieved passage ignores a core evaluation surface for retrieval-augmented civic helpers.
Exam caveat: separate retrieval metrics from generation metrics on the exam. A strong generator can hide a weak retriever until you measure passage relevance independently of answer polish.
Operational check: for detour and closure questions, label a small set of queries with relevant chunks, score retrieval hit rate or rank quality on that set, and only then judge answer prose on correctly retrieved evidence before promoting the viaduct helper.