Before promoting a revised prompt template for the permitting chatbot, the team wants an automated test suite that compares the new prompt's outputs against a set of known-good reference answers, catching regressions before the change reaches residents. Which practice best supports this?
Select an answer to reveal the explanation.
Short Explanation
Shipping a revised prompt without checking its answers against known-good examples is a bit like proofreading a letter by only reading the envelope. Running the new prompt automatically against a curated set of reference question-and-answer pairs, and comparing what comes back, is what actually catches a regression before it reaches a resident asking about their permit.
Full Explanation
An automated prompt-testing framework runs a candidate prompt against a fixed set of reference questions with known-good expected answers, then compares the generated outputs against those references (using similarity scoring, exact-match checks, or a structured rubric) to flag regressions before the prompt is promoted. A single person's impression from a handful of ad hoc questions isn't repeatable, doesn't scale to catching subtle regressions across many scenarios, and leaves no record for the next reviewer to check against. Skipping pre-promotion testing and waiting for residents to report problems turns the production system into the test environment, which risks residents receiving wrong permitting guidance before anyone notices. Re-testing only the old prompt provides no information about how the new candidate behaves — it verifies nothing about the actual change being promoted. Scope caveat: reference answers need periodic review too, since permitting rules change over time and a stale reference set can flag a correct, updated answer as a false regression. Operational check: intentionally introduce a prompt edit known to cause a wrong answer on one reference question and confirm the automated test suite catches it before promotion.