During evaluation, Meridian Line's maintenance-triage agent responds to a query about a crane's hydraulic pump with a specific part number that does not appear anywhere in the maintenance logs the retrieval step returned for that query. What has occurred, and what evaluation approach would have caught it?
Select an answer to reveal the explanation.
Short Explanation
The retrieval step did return logs, the model just stated something those logs never said. That's a hallucination specifically, and the way to catch it is a groundedness check that compares each claim in the answer against what was actually retrieved.
Full Explanation
The scenario describes retrieval succeeding, logs were returned, but the model generating a specific factual claim, a part number, that is not supported by anything in that retrieved content, which is the definition of a hallucination or ungrounded response, distinct from a retrieval failure where nothing relevant comes back at all. The evaluation approach that catches this specifically is a groundedness or faithfulness check, which compares the claims made in the generated response against the retrieved passages and flags claims that cannot be traced back to the source material. Latency regression is a timing concern with no relationship to whether a stated fact is supported by retrieved content, an agent can respond instantly with a fabricated part number. An empty-retrieval failure is a different, narrower failure mode where the retrieval step itself returns nothing relevant, but the scenario states logs were returned, the failure occurred in generation, not retrieval, so a check for empty results would not catch this case. A cost overrun concerns token consumption and spend, unrelated to whether the content of the response is factually supported by what was retrieved. Exam framing: hallucination detection specifically means checking generated claims against retrieved evidence, which is a different check than retrieval-empty-result handling, latency, or cost.