Nightly CI must run a frozen civic scenario suite before any agent prompt change merges. What evaluation practice is this?
Select an answer to reveal the explanation.
Short Explanation
Think of the frozen scenario suite as the city’s seatbelt for prompt edits—CI runs it before merge. That’s an automated eval gate, not “ship it, I feel good.” Deleting failing cases to go green is cheating the test.
Full Explanation
Nightly CI that runs a frozen civic scenario suite before any agent prompt change merges is automating an evaluation pipeline as a merge gate that blocks regressions on a fixed municipal scenario suite. The frozen suite is the city’s seatbelt for prompt edits: changes cannot land unless scored behaviors still pass. That MLOps-aligned practice protects 311 and other civic agents from silent quality drops driven by well-intentioned prompt tweaks.
Manual spot checks by whoever is free after lunch with no recorded suite are non-repeatable and miss regressions. Promoting prompt edits straight to production based on author confidence skips evidence. Deleting failing scenarios so the pipeline stays green falsifies the quality signal and trains the organization to hide breakage.
Exam caveat: the scenario suite must stay frozen for gating purposes; legitimate scenario updates need their own reviewed change process, not silent deletion when red. Operational check: submit a deliberate prompt regression in a branch, confirm CI fails the frozen civic suite and blocks merge, restore the good prompt, confirm green, and audit that failing scenarios were fixed or waived with review—not deleted to go green.