A public-health records team is deciding how to evaluate their records-lookup agent. One option scores only the final response returned to the requester; another inspects the full reasoning trajectory, including which tools the agent called and in what order. When is trajectory evaluation the more appropriate choice over final-response-only scoring?
Select an answer to reveal the explanation.
Short Explanation
Grading only the final answer is like checking a delivery only by whether the package arrived, never asking if the driver took a wildly wrong route to get there. Trajectory evaluation opens the route: it catches the agent that got the right records lookup by accident, or through a path that won't hold up under a different query.
Full Explanation
Trajectory evaluation examines the sequence of tool calls and intermediate decisions an agent makes, which matters most when a correct final answer could mask an unreliable or inefficient path — for example, a records-lookup agent that stumbles onto the right patient record after several incorrect queries would pass a final-response check while hiding a real reliability problem that could surface differently on the next request. Final-response scoring alone cannot distinguish a sound process from a lucky one, since it only sees the outcome. Reducing test-case count is unrelated to what dimension is being scored — trajectory evaluation typically requires more instrumentation, not less, so it doesn't shrink the evaluation set. Trajectory data being a more stable storage format is not a real evaluation consideration; format stability doesn't determine which evaluation approach is more informative. Trajectory evaluation is by definition dependent on access to the agent's reasoning traces, so an evaluation setup lacking that access cannot use it at all — the opposite of what makes it a fit. Scope caveat: trajectory evaluation requires success criteria defined for the intermediate steps, not just the end state, or there is nothing concrete to score the path against. Operational check: pull a transcript where the final answer was correct and confirm the trajectory evaluation flags it if the tool sequence deviated from the golden path.