An engineering team is evaluating where in their SDLC to introduce agentic automation. They track four metrics: story-point velocity, defect escape rate, mean time to merge (MTTM) for pull requests, and on-call incident frequency. After a six-week pilot, agent-assisted code review reduced MTTM by 40% but defect escape rate increased by 12%. Agent-assisted test generation had no measurable velocity impact but reduced defect escape rate by 18%. Which conclusion BEST describes where agentic automation is currently adding genuine value?
Select an answer to reveal the explanation.
Short Explanation
You can think of MTTM like checkout speed at a grocery store — faster lines feel great until the cashier starts letting shoplifters through. A 40% faster merge time paired with a 12% higher defect escape rate means the review agent is creating downstream rework. Test generation, by contrast, directly cuts defects reaching customers — the metric that actually matters most.
Full Explanation
This question tests whether candidates understand the difference between proxy metrics, such as velocity and merge speed, and outcome metrics, such as defect escape rate and incident frequency. Agent ROI must be measured against outcomes that matter to the business, not just throughput.
The conclusion that agent-assisted code review is succeeding because throughput improvements always outweigh quality trade-offs is wrong because throughput improvements do not always outweigh quality trade-offs. The 40% reduction in mean time to merge sounds impressive, but if the review agent is rubber-stamping pull requests and allowing 12% more defects to escape into production, those defects will create rework, incidents, and customer impact that cost far more than the time saved. Speed without quality is anti-value.
The conclusion that agent-assisted test generation is adding genuine value because it improves the metric most closely tied to customer outcomes, even without velocity gains, is correct. Test generation did not improve velocity, but a reduction in defect escape rate by 18% is directly tied to customer experience and production stability. Defects that don't escape don't become incidents, don't trigger on-call responses, and don't require emergency patches. The ROI is real even if it's not visible in story points. The absence of velocity improvement is simply neutral, not negative.
The conclusion that neither agent is adding value because neither produced improvements across all four metrics simultaneously applies a too-strict standard. Requiring all four metrics to improve simultaneously before declaring value ignores that individual SDLC interventions target specific stages. The test agent is not designed to move story-point velocity; it's designed to catch bugs earlier.
The conclusion that both agents are adding value equally, and the team should deploy both to production immediately, is wrong because deploying an agent that demonstrably increases defect escape rate is harmful. The review agent needs root-cause analysis before any production deployment — it may need calibration, a stricter escalation policy, or human checkpoints before final merge approval.