A senior engineer is reviewing the autonomy configuration for a code-review agent that has been in production for six months. The agent started at a conservative autonomy level but the team wants to expand its scope. Which set of signals MOST appropriately indicates that the agent is ready for expanded autonomy?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Volume and vibes are not readiness signals — they're popularity signals. A low override rate just means engineers got used to clicking through. Autonomy expansion requires a composite signal: processed volume, quality confirmed by structured measurement, and a clean safety record (no missed findings that caused incidents). Model upgrades change capability in the lab but don't change field-validated trust in your specific use case.
Full explanation below image
Full Explanation
Expanding agent autonomy requires multi-dimensional evidence, not a single metric. Each signal type captures a different aspect of readiness.
Option A (1,000 PRs + no overrides) is weak because it only measures volume and one behavioral signal. Zero overrides sounds positive, but it could mean: engineers find the agent's comments high-quality, OR engineers are ignoring agent comments entirely, OR the agent is producing comments so cautious they never require correction. Without a quality measurement, you cannot distinguish these interpretations. Volume alone tells you the agent is used, not that it is accurate.
Option B is correct because it is a composite signal across four dimensions: (1) Volume — sufficient sample size to observe edge cases and variance. (2) Low override rate — engineers are accepting agent conclusions, not routinely correcting them (behavioral signal). (3) Quality surveys — structured measurement of perceived correctness, completeness, and actionability. This distinguishes 'not overridden' from 'actually helpful.' (4) Zero safety/compliance false-negative incidents — the most critical gate. If the agent missed a security vulnerability or compliance violation that caused a production incident, the volume and survey scores are irrelevant. This safety signal must be clean over a meaningful observation window (ideally 90 days minimum). Only when all four dimensions are positive does the composite indicate readiness.
Option C (positive retrospective sentiment) is anecdotal and subject to recency bias, social desirability bias, and confirmation bias. Engineers who like the agent's general personality may not accurately report whether it is catching important issues.
Option D (model upgrade + benchmark improvement) is laboratory performance, not field performance. A model that scores higher on coding benchmarks may still produce worse results in your specific context (codebase conventions, security policies, domain-specific requirements). Benchmark improvement is a reason to re-evaluate, not a readiness signal itself.