You are designing an agent that automatically triages GitHub Issues, assigns labels, and routes them to the correct project board column. Before implementing the agent, you need to define its success criteria. Which definition most completely captures both the output contract and measurable success?
Select an answer to reveal the explanation.
Short Explanation
You need success criteria that name the artifact, its schema, and how you measure correctness. The triage record with validated labels, destination column, confidence score, and reasoning trace, plus 95% label accuracy against a human-reviewed validation set, does that. Closing issues, avoiding exceptions, or posting comments doesn't prove the triage is right.
Full Explanation
Defining success criteria and output contracts is foundational in agent architecture. Without them, you cannot validate behavior, compare agent versions, or detect regressions. Good definitions combine three elements: structured output artifact with defined schema, measurable quality criteria, and reference benchmark or validation method.
The definition that names the triage record per issue containing assigned labels from validated taxonomy, destination column, confidence score, and reasoning trace — with at least 95% label accuracy measured against human-reviewed validation set — is correct. It specifies the output artifact (triage record per issue), its schema (labels, destination column, confidence score, reasoning trace), and measurable quality bar (95% label accuracy vs. human-reviewed validation set). This is complete output contract — downstream system or human reviewer knows exactly what to expect and how to evaluate it.
The definition that says the agent successfully closes all open issues within 24 hours of their creation defines success in terms of throughput (closing issues) rather than quality. Closing all issues within 24 hours could mean auto-closing them without review — destructive outcome that satisfies metric but violates intent. Activity is not the same as correctness.
The definition that says the agent runs without throwing exceptions and completes all API calls successfully defines success as 'no exceptions thrown.' This is operational health check, not success criterion. An agent can complete every API call successfully while producing incorrect labels on every issue. Execution health and output quality are separate concerns.
The definition that says the agent processes every new issue within 5 minutes of creation and posts comment confirming triage specifies latency (5-minute SLA) and notification behavior (comment). These are valid SLA requirements, but they say nothing about accuracy or structure of triage decision itself. Speed with wrong labels is worse than slow with right ones.
The exam objective is: 'Define inputs, outputs, and success criteria for agents.' The output contract must be schema-driven and quality must be measurable against defined benchmark — not inferred from execution health or throughput alone.