An agent scores 95% on its evaluation benchmark before deployment. In production, the team finds the agent is performing poorly on edge cases involving legacy Python 2 codebases. Investigation reveals the evaluation dataset contained only Python 3 examples. What does this incident illustrate?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Scoring 95% on a Python 3 benchmark tells you exactly nothing about Python 2 performance — you measured the wrong thing. Eval dataset distribution is not a formality; it must mirror the actual production input distribution, including edge cases, legacy formats, and the full variety of real-world inputs the agent will encounter.
Full explanation below image
Full Explanation
Evaluation dataset distribution mismatch is a fundamental validity problem in agent evaluation: when the benchmark does not represent the production input distribution, high benchmark scores are misleading. The agent is being evaluated on a different task than the one it will perform in production.
The fix requires understanding the production input distribution before building the evaluation dataset. For this scenario: What percentage of production codebases are Python 2? What percentage are Python 3? Are there other language versions or variants? The evaluation dataset should sample inputs proportional to production distribution, with deliberate over-sampling of rare but important edge cases (like legacy Python 2) to ensure the agent's performance on those cases is explicitly measured.
This is distinct from a small sample size problem — even 10,000 Python 3-only examples would not reveal Python 2 performance issues. The issue is distribution, not volume.
Option A (model does not support Python 2) is an empirical claim that was not established by the scenario. Many LLMs were trained on Python 2 code and can process it. The problem is that the evaluation never tested this capability.
Option C (lower confidence threshold) is unrelated to distribution mismatch. The agent is not being overly cautious about Python 2 tasks — it is performing poorly on them. Lowering confidence threshold would cause it to attempt them with lower confidence, not to perform better.
Option D (sample size below 1000) assumes the problem is statistical significance. The problem is systematic exclusion of a category of input, not insufficient sample size. 10,000 Python 3 examples would still produce the same production failure.
Evaluation dataset must mirror production input distribution, including edge cases.