You are building an evaluation dataset for a coding agent that fixes GitHub Issues. The dataset must allow you to objectively measure whether the agent's fix is correct. What makes an evaluation example in this dataset complete and objectively scoreable?
Select an answer to reveal the explanation.
Short Explanation and Infographic
An evaluation example without an automated verifier is like a math problem without an answer key — you can collect answers but you need a human to grade every single one. The automated test is the answer key: it tells you objectively whether the agent solved the problem, at scale.
Full explanation below image
Full Explanation
A complete, objectively scoreable evaluation example for a code-fixing agent requires four components: (1) the input (issue description), (2) the environment (repository state), (3) a ground truth reference (the expected correct fix), and (4) an automated verifier (a test that passes only when the issue is correctly resolved). The automated verifier is the critical element for scale — it allows thousands of examples to be scored without human review of each one.
Option A (human judgment of 'looks correct') is subjective, does not scale, and cannot be run programmatically in an automated evaluation pipeline. Different humans may judge the same fix differently, making the evaluation noisy.
Option C (three example fixes from different developers) shows the solution space but does not provide a machine-checkable oracle. The agent's fix might be equivalent to one of the examples but syntactically different, making automated comparison unreliable. It also requires curating multiple correct solutions per example, which is expensive.
Option D (severity rating) is a prioritization tool for issue triage, not an evaluation completeness criterion. It says nothing about whether the agent's fix is correct.
The gold standard evaluation dataset for coding agents (exemplified by benchmarks like SWE-bench) includes the repository state, the issue description, and an automated test oracle that definitively verifies fix correctness. This enables reproducible, scalable, objective evaluation.