A team evaluates their code-generation agent solely by tracking whether the generated code passes all unit tests. Over three months, the test pass rate remains above 95%, but developers repeatedly report that the generated code is unnecessarily complex, uses deprecated APIs, and is difficult to read. The evaluation metric is not capturing this feedback. What is the team missing?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Measuring only whether the code passes tests is like grading a restaurant solely on whether the food reaches the table — it might arrive cold, oversalted, and looking terrible, but hey, it arrived. Test pass rate is a floor, not a ceiling. A full evaluation suite includes qualitative signals: readability scores, complexity metrics, API currency checks, and developer satisfaction.
Full explanation below image
Full Explanation
Test pass rate is a necessary but insufficient evaluation signal for a code-generation agent. It measures functional correctness — does the code do what it is supposed to do — but it says nothing about how well the code does it. Code can pass every test while being: - Unnecessarily complex (high cyclomatic complexity) - Using deprecated APIs (technically works but creates technical debt) - Difficult to read and maintain - Not following team conventions or style guides
Qualitative evaluation signals that complement functional metrics include: - Developer satisfaction surveys (thumbs up/down after each code generation) - Automated code complexity analysis (e.g., cyclomatic complexity scores) - Linting rules for deprecated API usage - Readability scoring tools - Peer code review ratings - Time-to-understand metrics from developers
Combining quantitative metrics (test pass rate, complexity scores) with qualitative feedback (developer ratings, review comments) gives a complete picture of agent output quality.
Option A (more unit tests) improves functional coverage but does not address the qualitative dimensions the developers are reporting.
Option C (larger model) may improve output quality, but without qualitative evaluation signals you have no way to measure whether the improvement is happening or which dimensions improved.
Option D (replace test pass rate with manual review) throws away a valuable objective signal and replaces it with a subjective one that is expensive and hard to scale. The correct approach is to add qualitative signals alongside test pass rate, not replace it.