A team has built an autonomous coding agent that writes, tests, and iterates on code. They want to evaluate whether the agent is ready for production use on real engineering tasks. Which evaluation methodology gives the most reliable production readiness signal?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — c is correct because production readiness evaluation must use tasks that reflect real production conditions: actual issue tickets expose the agent to ambiguous requirements, incomplete context, and domain-specific codebases that synthetic benchmarks do not capture. Task completion rate measures whether the agent can navigate real-world complexity; test pass rate provides objective correctness measurement; human review catches subtle issues in edge cases.
Full explanation below image
Full Explanation
C is correct because production readiness evaluation must use tasks that reflect real production conditions: actual issue tickets expose the agent to ambiguous requirements, incomplete context, and domain-specific codebases that synthetic benchmarks do not capture. Task completion rate measures whether the agent can navigate real-world complexity; test pass rate provides objective correctness measurement; human review catches subtle issues in edge cases. B is wrong because isolated coding challenges (LeetCode-style) measure algorithmic problem-solving capability, not the ability to navigate ambiguous requirements, understand existing codebases, or handle the open-ended nature of real engineering tasks. A is wrong because 10 outputs reviewed by engineers is statistically insufficient and subject to reviewer bias — it does not cover the variance in real task difficulty. D is wrong because token efficiency (tasks per million tokens) measures cost, not quality — an agent can be token-efficient while producing incorrect code.