An organization is designing an automated evaluation pipeline for agent-generated code changes. The pipeline collects four signals: (1) unit test pass rate across the existing test suite, (2) sentiment analysis of reviewer comments on the PR, (3) lines of code changed in the diff, and (4) total CI build time in seconds. Which signal is the MOST reliable automated indicator of whether the agent-generated code is functionally correct?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of it like this: if you want to know whether a machine is making good parts, you measure the parts — you do not measure how long the machine took to run, how much scrap it produced, or poll the shift supervisor's mood. The correct answer is A: unit test pass rate is objective, deterministic, and directly measures correctness. Sentiment analysis (B) is subjective, LOC (C) is a proxy with no correctness correlation, and build time (D) measures performance not behavior.
Full explanation below image
Full Explanation
Automated evaluation of agent-generated code must rely on signals that are objective, deterministic, and directly tied to the property being measured. For functional correctness, that means executing code against known inputs and verifying expected outputs.
Why A is correct: Unit tests are executable specifications. Each test asserts that a specific input produces a specific output — and the pass/fail result is unambiguous, repeatable, and independent of human judgment. For agent-generated code evaluation at scale, unit test pass rate provides the most reliable automated correctness signal because: (1) it directly exercises the code's behavior, (2) a passing test suite means the code satisfies every previously defined behavioral contract, and (3) the metric is fully deterministic — the same code will produce the same result on every run.
Why B is wrong: Sentiment analysis of reviewer comments is inherently subjective. Reviewers may write neutrally worded comments about serious bugs ('might want to consider handling the null case') or negatively worded comments about minor style issues ('I really hate this naming convention'). Sentiment scores measure tone, not correctness. Additionally, reviewer feedback is human-paced and may arrive hours or days after the PR is submitted, making it unsuitable as a real-time automated gate.
Why C is wrong: Lines of code changed is a proxy metric frequently cited in change-risk discussions, but it has no direct correlation with correctness. A 5-line change can introduce a critical off-by-one error in a sorting algorithm; a 500-line change can correctly implement a well-specified feature with full test coverage. An agent optimizing for small diffs could produce very small, very incorrect changes.
Why D is wrong: CI build time measures compilation speed and dependency resolution, not code behavior. An agent that adds an efficient new library might increase build time while improving code correctness. An agent that introduces a logic error in a pure function will not change build time at all. Build time is a performance signal, not a correctness signal.