A product manager deployed a custom support-triage agent used by 60 reps that tags intent and drafts macros. Quality slipped this week with wrong tags. Which method should be used to review custom agent activities and diagnose behavior?
Select an answer to reveal the explanation.
Short Explanation
You've got game tape for your agent, so don't guess why tags slipped. Check your session logs for prompts, tool calls, and feedback and you'll spot the pattern fast.
Full Explanation
The correct method is direct activity review because diagnosing wrong intent tags across 60 reps requires evidence from actual executions, and session logs paired with prompts, tool calls, and user feedback reveal whether the failure stems from ambiguous instructions, missing context, faulty retrieval, or misapplied tools by showing patterns across time, users, and intents. Reviewing aggregate accuracy dashboards and satisfaction scores is standard for ongoing performance monitoring but fails for diagnosis because averages and sentiment obscure the step-by-step reasoning and tool inputs that explain why a specific tag was chosen. Auditing instruction version history and permission scopes is sound change control and useful if a bad deployment is suspected, but fails here because it verifies what configuration should run rather than examining what the agent actually did in failing sessions. Sampling tagged tickets to retrain the classifier is a sound improvement practice for long-term accuracy, but fails as the initial diagnostic step because it jumps to remediation without first establishing root cause from activity traces. On exams, beware stems that ask to review activities and diagnose behavior, which signal observability and trace analysis rather than high-level metrics, governance audits, or retraining workflows. In practice, filter recent sessions for incorrect tags, compare prompt versions, retrieved context, tool arguments, and rep corrections side by side to isolate repeatable failure patterns before changing instructions or models.