A team evaluates their code-generation agent solely by tracking whether the generated code passes all unit tests. Over three months, the test pass rate remains above 95%, but developers repeatedly report that the generated code is unnecessarily complex, uses deprecated APIs, and is difficult to read. The evaluation metric is not capturing this feedback. What is the team missing?
Select an answer to reveal the explanation.
Short Explanation
You're measuring only whether code passes tests like grading the restaurant solely on whether food reaches table — it might arrive cold, oversalted, and looking terrible, but hey, it arrived. Test pass rate is floor, not ceiling. You need qualitative signals like developer satisfaction ratings, code complexity scores, and readability reviews.
Full Explanation
Test pass rate is necessary but insufficient evaluation signal for code-generation agent. It measures functional correctness — does code do what it is supposed to do — but it says nothing about how well code does it. Code can pass every test while being: - Unnecessarily complex (high cyclomatic complexity) - Using deprecated APIs (technically works but creates technical debt) - Difficult to read and maintain - Not following team conventions or style guides
Qualitative evaluation signals that complement functional metrics include: - Developer satisfaction surveys (thumbs up/down after each code generation) - Automated code complexity analysis (e.g., cyclomatic complexity scores) - Linting rules for deprecated API usage - Readability scoring tools - Peer code review ratings - Time-to-understand metrics from developers
Combining quantitative metrics (test pass rate, complexity scores) with qualitative feedback (developer ratings, review comments) gives complete picture of agent output quality.
The suggestion to increase the number of unit tests improves functional coverage but does not address qualitative dimensions developers are reporting.
The suggestion to upgrade the agent's model to larger model may improve output quality, but without qualitative evaluation signals you have no way to measure whether improvement is happening or which dimensions improved.
The suggestion to replace unit test pass rate entirely with manual code review throws away valuable objective signal and replaces it with subjective one that is expensive and hard to scale. The correct approach is to add qualitative signals alongside test pass rate, not replace it.