A platform team is building an agent to automate the migration of 500 microservices from Java 8 to Java 17. Before deploying the agent, the team needs to define what a successful outcome looks like. The project manager suggests: 'The outcome is successful when all 500 services have been migrated.' An architect argues this definition is incomplete. Which outcome specification would be MOST appropriate for evaluating the agent's performance?
Select an answer to reveal the explanation.
Short Explanation
You need an outcome that proves the services actually work under Java 17, not just that files changed. Require clean compiles, passing tests, no new critical CVEs, and successful staging deployments. That gives you objective, external proof instead of a circular claim.
Full Explanation
Outcome specification for agentic tasks must be externally verifiable, multi-dimensional, and failure-distinguishing. The project manager's definition ('all 500 services migrated') is circular: it makes 'migrated' the success criterion without defining what migrated means. This allows an agent to game the metric by making superficial changes.
The most appropriate specification is the one that requires all 500 services to compile without errors under Java 17, existing test suites to pass at a ≥98% rate, no new critical CVEs introduced by dependency changes, and each service's CI pipeline to confirm a successful build and deployment to staging. It uses four dimensions:
Functional correctness: services compile without errors under the new runtime. Compilation failure is an objective, binary test — no ambiguity. Behavioral preservation: existing test suites must pass at a high rate (≥98%). This ensures the migration did not break existing functionality. The 2% tolerance accounts for flaky tests pre-existing before the migration. Security integrity: dependency upgrades that accompany Java 17 migrations can introduce or resolve CVEs. Checking for new critical CVEs ensures the migration does not inadvertently worsen the security posture. Deployment validation: CI pipeline success to staging confirms the artifact not only compiles but also packages, deploys, and passes integration-level checks in a representative environment.
The pom.xml-only definition measures an agent action (it wrote a file) not an outcome (the service works). Services with corrupted pom.xml files might still commit successfully. The project-tracking self-attestation definition is fundamentally flawed: allowing the agent to declare its own success creates a metric that the agent can trivially satisfy without performing real work, and removes the external verifiability requirement. The migration-report definition measures documentation output, not migration quality. Detailed reports of incorrect changes are worthless. The exam principle is that outcome specifications must be based on independently verifiable evidence, not agent-reported actions or artifacts.