A permits chatbot team needs a fair way to judge answer quality before each release. What evaluation asset should they build?

Building a labeled test set of anonymized resident questions mapped to expected answers is the fair pre-release evaluation asset for a permits chatbot. Labels encode what a correct, policy-aligned answer should contain (fees, timelines, required forms, refusal cases). Anonymization protects resident privacy while preserving realistic municipal language. GenAIOps then scores each release against that set instead of gambling on a single lucky production chat after go-live.

Building a versioned archive of anonymized resident chats for manual spot checks before release fails because anonymized but unlabeled chats with ad hoc manual skimming give no ground-truth expected answer to score against — spot checks are unscalable and inconsistent, not a fair, repeatable release gate. Building a labeled regression set of training questions mapped to expected permit answers fails because testing against the model's own training questions mainly checks whether it memorized those exact items, not whether it can correctly answer the broader range of real resident questions it will actually face. Building a labeled test set of synthetic resident questions mapped to expected permit answers fails because machine-generated questions can miss the real phrasing, edge cases, and local ordinance quirks that anonymized actual resident questions capture, weakening how well the eval set predicts production quality.

Exam caveat: Labeled sets drift as ordinances change; schedule corpus refresh with code updates. Keep PII out of labels and storage, and version the dataset alongside prompts so scores remain comparable across releases.

Operational check: Confirm the permits eval set exists with question-to-expected-answer mappings; run it in CI on every prompt or model change; track failing items with owners; scrub logs used to grow the set; and require a passing threshold before Foundry production promotion.