A 311 program is deciding what counts as an adequate hallucination-detection safeguard for a GenAI assistant that answers resident questions about municipal services. A vendor argues that the underlying model's strong general accuracy reputation is sufficient assurance on its own. What should program leadership require instead?

A concrete hallucination-detection control, such as grounding the assistant's responses in verified municipal source documents and flagging low-confidence or unconfirmed answers, gives leadership an actual mechanism to catch fabricated information before it reaches a resident, rather than relying on a general impression of the model's quality. Accepting the vendor's reputation for accuracy as sufficient assurance ignores that even strong general-purpose models can still generate fabricated specifics, like an incorrect fee amount or a nonexistent permit requirement, when asked about narrow local municipal details the model wasn't specifically trained on. Shifting verification entirely onto residents defeats the purpose of deploying the assistant in the first place, since residents turn to 311 precisely because they don't already know the correct answer and can't be expected to independently fact-check it. A blanket disclaimer with no underlying technical detection mechanism provides legal cover in name only; it doesn't actually reduce the chance a resident receives and acts on incorrect information, it just documents that the possibility exists. A scope caveat: grounding and flagging reduce but don't eliminate hallucination risk entirely, so this control should be paired with ongoing monitoring rather than treated as a permanent fix. A concrete check: periodically sample the assistant's flagged low-confidence responses and confirm they're being routed to a human for review rather than silently delivered to the resident.