A team reports that their code-review agent passes all 200 scenarios in the evaluation suite during testing but frequently fails when deployed to real repositories. Failures cluster around unusual file structures, non-standard languages, and repositories with missing configuration files. What is the most appropriate corrective action?
Select an answer to reveal the explanation.
Short Explanation and Infographic
A driving test that only includes right turns will produce drivers who panic at left turns — that's exactly what an evaluation suite that only covers typical cases produces. When production failures cluster around edge cases not present in your eval suite, the suite itself is the gap. Expanding coverage to include real-world failure patterns is the correct evaluation-driven fix. The exam tests whether you recognize incomplete evaluation coverage as a distinct root cause.
Full explanation below image
Full Explanation
This question tests your ability to diagnose and remediate incomplete evaluation coverage — a failure mode where the evaluation suite is representative of happy-path scenarios but does not cover the diversity of real production inputs.
The correct answer is B. When production failures cluster around specific categories of inputs (unusual file structures, non-standard languages, missing configs), the right response is to sample those failure patterns and create corresponding evaluation scenarios. This is called evaluation-driven tuning: you use production failure data to expand your eval suite, then iterate on the agent until it handles those cases correctly. This ensures the evaluation suite actually reflects the distribution of inputs the agent encounters in production.
Option A is incorrect. Timeout increases address performance constraints, not capability gaps. The failures described are about the agent not knowing how to handle certain inputs, not about running out of time.
Option C is incorrect. Instructing the agent to skip unrecognized files treats the symptom rather than the root cause and degrades the agent's usefulness on real-world repositories. It would also silently ignore failures rather than correcting the agent's behavior.
Option D is incorrect. Deploying a separate agent for non-standard repositories is an architectural workaround that adds operational complexity and maintenance burden. It also sidesteps the evaluation problem — the second agent would have the same evaluation coverage gap unless specifically addressed. The correct approach is to fix the evaluation suite and improve the primary agent's handling of the problematic scenarios.
The key principle here is that evaluation suites must be continuously updated as production failure patterns are discovered. A static eval suite that was built before deployment will drift from reality over time.