A team wants to evaluate the quality of pull request feedback generated by their code review agent. They need signals that can be tracked over time and used to detect regressions in agent quality. A data scientist proposes using only quantitative signals (e.g., number of comments per PR, average comment length). A senior engineer argues that qualitative signals are also necessary. Which evaluation signal design BEST captures both dimensions for ongoing agent quality assessment?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Using only comment count to evaluate a code reviewer is like rating a doctor's performance by how many words they write in patient notes. Quantity without quality misses everything that matters. The best evaluation systems run two instruments in parallel: the objective meter (counts, rates, latency) and the human thermometer (satisfaction surveys, expert sampling). Together they catch both regression types.
Full explanation below image
Full Explanation
Effective evaluation signal design for agentic systems requires a portfolio of signals, not a single measure. This is because different failure modes are visible to different instruments:
Quantitative signals catch measurable regressions: - Critical issue detection rate: If the agent misses SQL injection vulnerabilities, this rate drops — detectable without human review - False positive rate: Too many spurious warnings causes developer fatigue and degrades trust — measurable against a labeled dataset - Latency: If processing time increases from 2 to 15 minutes, something changed in the agent's execution — measurable automatically
Qualitative signals catch semantic regressions that metrics miss: - Structured developer surveys (Likert scale: 'Was this feedback actionable? 1-5') provide a human-grounded quality signal that quantitative metrics cannot replicate. Even if the agent posts the same number of comments at the same rate, those comments might become less actionable — only developers can detect this. - Expert review sampling: A senior engineer reviewing a 5% random sample of agent outputs provides a ground-truth quality check against which the automated signals can be calibrated
Baseline values and regression thresholds are critical — without them, you know current values but cannot detect change over time.
Option A (quantitative only) creates metric blindspots: an agent that generates the same comment count but with increasingly vague or irrelevant content will pass all quantitative metrics while degrading in actual value. Option C (qualitative only) cannot be tracked programmatically or at scale — surveying developers on every PR is impractical for large teams. Option D (single LLM judge) replaces one uncertain evaluator (the primary agent) with another uncertain evaluator (a judge LLM) — this introduces systematic LLM bias and cannot be independently verified. The exam principle: use both quantitative (automatable) and qualitative (human-grounded) signals together.