A fire-public-info agent answers evacuation-zone questions grounded in GIS bulletins. Which quality metric best checks that answers address the resident’s location-critical ask?

Relevance scoring against expected answers for the resident’s evacuation-zone question is correct because relevance metrics ask whether the generative response actually addresses the user’s intent, not merely whether the answer scores well on similarity for wording overlap with expected answers. For a municipal fire-public-info agent grounded in GIS bulletins, the location-critical ask is “which evacuation zone applies to me?” Relevance evaluation compares model answers to gold GIS-grounded expectations for that address or zone, surfacing location-mismatched advice before residents act on it. That metric aligns GenAIOps quality gates with life-safety civic outcomes rather than similarity scoring alone.

Similarity scoring fails because wording overlap with expected answers can still hide the wrong zone, wrong shelter, or outdated bulletin—exactly the failure mode that harms residents during evacuations. Groundedness scoring against GIS bulletin sources is a source-faithfulness check, not a correctness signal for evacuation advice; a grounded response can still point to the wrong zone. Completeness scoring against expected answers checks coverage of the expected request, but full coverage alone does not confirm the answer addresses the resident’s specific location-critical ask; a complete response can still point to the wrong zone.

Exam caveat: distinguish relevance (intent match), similarity (wording overlap), groundedness (source faithfulness), and completeness (request coverage). The stem names a location-critical resident ask, so relevance against expected evacuation-zone answers is the primary quality check.

Operational check: build an eval set of address-to-zone gold pairs from current GIS bulletins, score agent responses with a relevance rubric or automatic similarity-to-expected metric, fail promotion when zone-mismatch rates exceed the fire department’s risk threshold, and re-run after each bulletin or index refresh.