An agent that automatically triages GitHub issues fails on 15% of runs. The failures manifest as the agent marking issues as 'needs more info' when it should have escalated them as 'critical.' A developer wants to determine whether the root cause is: (A) the agent's reasoning is flawed, (B) the tool that queries issue metadata is returning incomplete data, or (C) the agent's context about severity classification is outdated. Which diagnostic approach would MOST efficiently distinguish between these three failure classes?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Guessing which component is broken without looking at the evidence is like changing spark plugs, filters, and tires all at once to fix an engine misfire — wasteful and uninformative. Execution traces are the diagnostic readout: the tool call I/O tells you if the data was good, and the reasoning trace tells you if the agent thought about it correctly. You find the break before you apply the fix.
Full explanation below image
Full Explanation
Efficient root cause identification in agentic systems requires examining the execution trace at the specific failure point, not trying different interventions blindly. A well-structured execution trace captures:
For distinguishing tool failure (class B): - The exact input sent to the metadata query tool (request parameters, issue ID) - The exact output returned by the tool (fields present, values returned) - If the output is missing fields like severity_labels, affected_users, or linked_incidents, the tool is the failure point
For distinguishing reasoning failure (class A): - The agent's reasoning chain after receiving the tool output - If the tool returned complete data but the agent still classified the issue as 'needs more info,' the reasoning logic is the failure point - Specific reasoning traces might reveal the agent is applying an incorrect heuristic or weighting labels incorrectly
For distinguishing context failure (class C): - The agent's reference to its severity classification criteria in its reasoning - If the agent cites outdated classification thresholds (e.g., 'critical = >1000 affected users' when the current policy is '>100 affected users'), the context is stale
Option A (different LLM model) cannot distinguish the three failure classes because all three failure classes would produce the same symptom on a different model (wrong classification). If model B also gets it wrong, you still don't know why. If model B gets it right, you only know it's model-specific but not whether it's reasoning, tool, or context. Option C (adding examples to system prompt) is a remediation for class C without first confirming C is the cause. If the root cause is class B (bad tool data), improving the system prompt will not help. Option D (synthetic test set) is useful for measuring accuracy but not for diagnosing which component caused specific failures.