English scores on the city’s 311 agent rose, but Spanish task scores dropped. What continuous benchmarking gate should block promotion?
Select an answer to reveal the explanation.
Short Explanation
A citywide bump in English doesn’t excuse a Spanish slide. Multi-slice gates that block promotion on language regressions keep continuous benchmarking fair for every resident.
Full Explanation
Continuous benchmarking for a municipal 311 agent must treat language slices as first-class promotion gates, not optional dashboards. When English task scores rise while Spanish scores fall, the equitable control is to block promotion of that build until the Spanish (and other language) slice regressions are remediated—even if the English-only average looks healthier. Aggregate English gains can mask service inequity for Spanish-speaking residents who rely on the same civic channel for permits, trash, and safety issues. Slice-aware gates make continuous evaluation a fairness mechanism, not a vanity metric.
Promoting whenever the English-only average improves fails by concept because an English-centric mean erases minority-language regressions and institutionalizes unequal quality of service. Shipping on emoji sentiment for English tickets fails because sentiment proxies are not task-success measures for ordinance retrieval, routing accuracy, or correct fee guidance—and they ignore non-English residents entirely. Disabling multi-language evals so inequitable regressions “never appear” fails because hiding slices does not remove harm; it only removes the city’s ability to detect and stop biased quality drops before they reach production.
Exam caveat: NCP-AAI continuous-benchmarking items often tempt candidates toward overall averages or soft UX signals. Prefer controls that gate release on protected or operational slices (language, channel, neighborhood) when the stem shows one slice improving while another regresses.
Operational check: in the city’s eval harness, define Spanish (and other language) slice score thresholds as hard promotion blockers; run the candidate build against those slices; if Spanish task success drops below the gate, fail the promote job, open a remediation ticket with failing examples, and only re-promote after slice scores recover on a recorded eval run.