A Copilot agent completes a feature implementation task. All 47 unit tests and 12 integration tests in the CI pipeline pass. A security scan run as a separate, optional step later reveals that the agent introduced a SQL injection vulnerability in a new database query method. How should the agent evaluation framework classify this outcome?
Select an answer to reveal the explanation.
Short Explanation
You don't call a fire-wall hole a warning just because the rooms look great. If the agent passes all tests but introduces SQL injection, you classify it as failure. Security scanning has to be a required gate, not an optional afterthought.
Full Explanation
Agent evaluation frameworks must treat security outcomes with at least the same severity as functional outcomes — and in practice, higher. A SQL injection vulnerability can expose an entire database to exfiltration or destruction; a failing unit test cannot. When an agent introduces a critical security flaw, the classification must be unambiguous. The correct classification is failure because security vulnerabilities introduced by agent-generated code are objective failures. The evaluation framework should require security scanning as a mandatory gate — not optional or post-hoc — so that a security finding blocks the same way a failing test blocks. The fact that 59 tests passed is irrelevant: tests validate behavior against known inputs; they do not probe for injection attack surfaces. Making security scanning optional creates a process gap where agents can consistently pass evaluation while systematically introducing exploitable code. The answer that calls it success with a warning is wrong because classifying a SQL injection as a warning and scheduling a follow-up task treats a critical security regression as a minor quality issue. In a production system, this vulnerability could be exploited before the follow-up is addressed. The evaluation framework exists precisely to prevent such gaps from reaching production. The answer that calls it partial success is wrong because proportional scoring that averages security findings against passing test counts is a dangerous anti-pattern. A single critical vulnerability has asymmetric impact — it can negate all the functional correctness in the world. Security findings are binary gates, not weighted averages. The answer that says the security scan result is likely a false positive is wrong because that reasoning is the most dangerous. SAST and dynamic security scanners detect vulnerability patterns that functional tests are structurally incapable of finding. A SQL injection vulnerability exists in how user input is concatenated into a query — functional tests typically use controlled, valid inputs, not adversarial injection payloads. Passing tests and being vulnerability-free are orthogonal properties. False positive rates for well-tuned security scanners on clear injection patterns are very low, and the right response to any security finding is investigation — not dismissal.