The charter-desk clause assistant cites a specific clause number in its answer, but a reviewer discovers the cited clause number does not exist anywhere in the charterparty text that was actually retrieved for that query. What class of failure is this, and how should the evaluation pipeline be built to catch it going forward?

This is a hallucination, specifically an ungrounded citation: the model referenced a clause number that was not in the context retrieved for the query, so the citation was invented rather than grounded in the source material. Build the evaluation pipeline to extract every citation from each generated answer and verify that it appears in the retrieved chunks; a fluent, plausible answer is not evidence that its citations are accurate. A latency failure concerns response timing or an index that may still be loading, but speed says nothing about whether the answer is grounded, and the scenario provides no sign of a loading delay. A guardrail failure means an input filter should have blocked an inappropriate, unsafe, or out-of-scope question before generation; this routine clause question does not call for that intervention. A schema failure concerns output shape, such as a mismatch between the expected type and the generated output, tracked through the MLflow model signature; it does not explain a fabricated clause reference. The exam caveat is that a clause number appearing somewhere in the charterparty is not enough if it was absent from the context retrieved for this query. As a practical check, log the retrieved chunks alongside each answer, extract its cited clause numbers, and fail the evaluation whenever a citation cannot be matched to that query’s retrieved context.