A developer wants to evaluate whether a code generation agent is producing production-ready code. They begin reviewing sample outputs but find themselves disagreeing with team members about which outputs are 'good.' The evaluations are inconsistent from reviewer to reviewer and from week to week. What is the MOST important first step before evaluating agent output?
Select an answer to reveal the explanation.
Short Explanation
You can't grade a test if there's no answer key. Inconsistent evaluations mean everyone's using a different mental rubric. Write down exactly what 'good' looks like—measurable, specific, agreed-upon criteria—before scoring a single output.
Full Explanation
Evaluation reliability begins with specification, not measurement. When evaluators disagree about what constitutes a 'good' output, the root cause is almost always the absence of explicit, shared evaluation criteria. Each reviewer is applying an implicit, personal definition of 'production-ready,' which will naturally vary between individuals and across time.
The correct first step is to define evaluation criteria before reviewing outputs. For a code generation agent, 'production-ready' might be specified as: passes all existing unit tests, includes docstrings on public functions, no hardcoded credentials, follows the project's style guide, and handles null inputs without crashing. These are concrete, verifiable, and reviewer-independent.
Well-defined criteria serve multiple purposes: - They enable consistent evaluation across reviewers and time periods - They make the evaluation scoreable and comparable - They reveal gaps between what the agent is optimizing for and what the team actually needs - They provide a feedback signal that can be used to improve the agent
Why the other approaches fail: - The automated-tests-only approach is useful but insufficient. Test pass rate measures functional correctness but misses readability, security, style, documentation, and maintainability — all components of production-readiness. Single-metric evaluation is almost always too narrow. - The single-reviewer approach reduces variability through constraint but doesn't fix the underlying problem. One reviewer with an undefined rubric will still be inconsistent over time, and single-reviewer evaluation doesn't scale or survive personnel changes. - The larger-sample-first approach reverses the correct order of operations. Collecting more outputs before defining success criteria means those outputs cannot be evaluated consistently — you'll just have more inconsistently evaluated samples.
Criteria definition is always the prerequisite step for any valid evaluation process.