After refreshing the storm-debris FAQ index, 311 wants to know whether answers still match resident questions well. What should they measure?

Measuring relevance and quality of generative FAQ answers against representative resident questions is what 311 should do after refreshing the storm-debris FAQ index. Index refreshes can improve document freshness yet also disturb chunk boundaries, embeddings, or retrieval rankings. Offline or staged evaluation with a labeled question set scores whether answers still address pickup rules, drop-off sites, bag limits, and eligibility with relevant retrieved context—not merely whether generated answers are grounded in the refreshed storm debris source documents.

Measuring groundedness of generated FAQ answers against the refreshed storm debris source documents checks source alignment, not whether the resulting answers still address resident questions well. Measuring recall of refreshed debris retrieval across representative storm cleanup topics checks retrieval coverage, but does not score whether the retrieved content actually answers the question. Assessing fluency and coherence of generated FAQ phrasing for resident readability is a readability check, not an assessment of answer relevance and quality after the index change.

Exam caveat: after RAG index refreshes, re-evaluate answer relevance and quality; source grounding alone is not a release gate for municipal FAQ bots. Operational check: run the pre-refresh evaluation set against the new debris index, compare relevance and groundedness scores to baseline, and gate public cutover on meeting the agreed 311 quality threshold for storm-debris intents.