A developer wants to evaluate whether a code generation agent is producing production-ready code. They begin reviewing sample outputs but find themselves disagreeing with team members about which outputs are 'good.' The evaluations are inconsistent from reviewer to reviewer and from week to week. What is the MOST important first step before evaluating agent output?
Select an answer to reveal the explanation.
Short Explanation and Infographic
You can't grade a test if there's no answer key. Inconsistent evaluations are a symptom of undefined success criteria — everyone is applying a different mental rubric. The first step is always to write down exactly what 'good' looks like: measurable, specific, agreed-upon criteria that every reviewer applies the same way before a single output is scored.
Full explanation below image
Full Explanation
Evaluation reliability begins with specification, not measurement. When evaluators disagree about what constitutes a 'good' output, the root cause is almost always the absence of explicit, shared evaluation criteria. Each reviewer is applying an implicit, personal definition of 'production-ready,' which will naturally vary between individuals and across time.
The correct first step is to define evaluation criteria before reviewing outputs. For a code generation agent, 'production-ready' might be specified as: passes all existing unit tests, includes docstrings on public functions, no hardcoded credentials, follows the project's style guide, and handles null inputs without crashing. These are concrete, verifiable, and reviewer-independent.
Well-defined criteria serve multiple purposes: - They enable consistent evaluation across reviewers and time periods - They make the evaluation scoreable and comparable - They reveal gaps between what the agent is optimizing for and what the team actually needs - They provide a feedback signal that can be used to improve the agent
Why the other options fail: - Option A (automated tests only) is useful but insufficient. Test pass rate measures functional correctness but misses readability, security, style, documentation, and maintainability — all components of production-readiness. Single-metric evaluation is almost always too narrow. - Option C (single reviewer) reduces variability through constraint but doesn't fix the underlying problem. A single reviewer with an undefined rubric will still be inconsistent over time, and single-reviewer evaluation doesn't scale or survive personnel changes. - Option D (collect more samples first) reverses the correct order of operations. Collecting more outputs before defining success criteria means those outputs cannot be evaluated consistently — you'll just have more inconsistently evaluated samples.
Criteria definition is always the prerequisite step for any valid evaluation process.