Before a new 311 agent build takes full traffic, ops wants proof it is not worse than last week. What continuous benchmarking practice fits?
Select an answer to reveal the explanation.
Short Explanation
You can protect residents from an unproven cutover by checking each build against a trusted baseline first. Shadow-scoring the release on last week’s gold 311 set keeps continuous benchmarking honest.
Full Explanation
Before a new 311 agent build takes full traffic, ops needs evidence that it is not worse than last week. Shadow-scoring every release against last week’s gold 311 set provides that continuous benchmark: it compares the candidate with a known-good baseline on fixed civic tasks, including success, faithfulness, latency, and refusals, without exposing all residents to an unproven cutover. Deploying each build immediately to all residents and relying on live usage instead of a benchmark set makes regressions a public incident by default and fails change safety. Comparing only against a coin flip to decide whether quality “feels lucky” is not sound evaluation. Skipping benchmarking because release notes sound confident substitutes confidence for evidence. Exam caveat: gold sets can drift as city policies and 311 taxonomies change, so refresh them on a cadence and keep them free of real exam brain-dump content. Shadow-scoring can be paired with canary percentages, but a canary without a gold comparison still lacks an absolute quality anchor. Practical check: automate scoring in CI/CD against the gold set, block full traffic on regressions beyond agreed deltas, store scorecards with build IDs, then ramp the build while monitoring live SLOs for surprises the set missed.