When the customs-compliance agent gives an incorrect answer, the evaluation team wants their harness to tell them whether the failure happened because Vector Search returned the wrong tariff clause, or because the LLM had the right clause in front of it but summarized it incorrectly. How should the evaluation pipeline be designed to make this distinction?

Attributing a failure to a specific pipeline stage requires scoring each stage independently: a retrieval-relevance metric checks whether the correct tariff clause was among the retrieved chunks, and a generation-faithfulness metric checks whether the final answer is actually supported by whatever was retrieved. Comparing the two per example tells the team directly which stage failed, retrieval never surfaced the right clause, or generation had it and mangled it. Scoring only end-to-end final-answer correctness is the classic trap: it is simpler to compute but collapses two distinct failure modes into one binary signal, leaving the team unable to tell whether to fix the chunking and index or the prompt and model without manual inspection of every failing case. Repeatedly swapping embedding models and re-running until the answer happens to come out correct is not evaluation at all, it is trial and error that produces no generalizable diagnosis and could accidentally mask a real generation bug behind a lucky retrieval result. AI Gateway payload logs record what was sent and returned at the endpoint boundary, which is useful for auditing traffic, but by itself it does not include the intermediate retrieval-relevance or faithfulness scoring needed to separate the two failure modes; that requires purpose-built evaluation metrics run against the traced intermediate steps, not just the raw request and response.