Each week the city wants proof the live ordinance QA LLM has not silently regressed versus last month. What practice fits?
Select an answer to reveal the explanation.
Short Explanation
You can catch silent regressions by rerunning a frozen ordinance QA suite and comparing the live deploy with last month’s build—like weighing the same truck on the same scale. If the scores slip, you have a clear signal to investigate.
Full Explanation
To show that the live ordinance QA LLM has not silently regressed versus last month, rerun a frozen ordinance QA suite and compare the current deploy with last month’s build. Using the same questions and scoring rubric is like weighing the same truck on the same scale: score changes are more likely to reflect model or prompt drift than a change in test difficulty. This comparison can surface silent regressions before ordinance guidance misleads staff or residents. Relying solely on informal staff opinions without a fixed suite gives you non-comparable anecdotes and can miss systematic failures. Refreshing QA questions weekly from recent cases can keep the evaluation representative of current usage, but changing the test set makes week-to-week scores harder to compare and does not by itself establish that the live deploy has stayed stable. Benchmarking only brand-new models and never retesting the live deploy leaves the production system unmonitored. Exam caveat: freeze both questions and labels; if ordinances change legally, version the suite and document the updates so policy changes are not mistaken for model failure. Practical check: store the frozen suite and last month’s model artifact in the registry, schedule weekly identical evaluations of the live deploy against that build, alert on score drops beyond a threshold, and hold promotion or trigger rollback review when the live system regresses.