A team is defining success criteria for a refactoring agent that will automatically update deprecated API calls across a large codebase. Which success criteria definition best incorporates both functional performance and Responsible AI safety constraints?
Select an answer to reveal the explanation.
Short Explanation
You don't just tell an agent to fix everything fast. You tell it what to do and what not to do. The right criteria keeps it in scope, preserves tests, flags ambiguity, and leaves an audit trail.
Full Explanation
Responsible AI success criteria for an autonomous agent are not just about functional outcomes — they must also encode behavioral constraints that bound the agent's authority and decision-making. Well-defined success criteria for this refactoring agent include multiple dimensions.
Functional success: the deprecated API calls are updated to their modern equivalents, and all existing test coverage is maintained after the changes.
Scope constraint: the agent only modifies files it has been explicitly authorized to touch. This prevents scope creep where an agent, in pursuit of its functional goal, modifies adjacent files that were not approved for change.
Uncertainty handling: when the agent encounters an ambiguous case (e.g., a deprecated call used in an unusual way that might require semantic understanding to safely refactor), it flags the case for human review rather than guessing. This is the 'know your limits' principle for autonomous systems.
Auditability: a change manifest that documents every file touched, every substitution made, and every case flagged provides the post-execution record required for compliance and debugging.
The definition that requires maintaining all existing test coverage, limiting changes to explicitly in-scope files, flagging ambiguous cases for human review, and producing a change manifest for post-execution audit is correct because it combines functional performance with Responsible AI safety constraints.
The definition that requires the agent to refactor 100% of deprecated API calls within the sprint timeframe is incorrect because 100% completion as the sole criterion creates incentive for the agent to make uncertain changes rather than flagging them, violating the 'flag ambiguity' safety constraint.
The definition that requires the agent to refactor deprecated API calls and improve overall code quality metrics by at least 20% is incorrect because code quality improvement is outside the authorized scope of a refactoring agent. An agent optimizing for quality metrics may make unintended changes to the codebase beyond its approved scope.
The definition that requires the agent to complete refactoring without introducing any new bugs as measured by the CI test suite is incorrect because 'no new bugs in CI' is a necessary condition but not sufficient. CI test suites often do not cover all production behavior. An agent could pass all tests while making scope-violating changes, failing to document its actions, or silently making uncertain substitutions.