Parks leadership wants monthly assurance the trail chatbot does not invent closed trails. Which audit approach best fits?

Parks leadership's monthly assurance that the trail chatbot does not invent closed trails is best served by hallucination auditing: sample bot answers and check invented or asserted trail names and closure claims against the authoritative corpus. Generative models can produce fluent falsehoods; only systematic comparison to ground-truth park data reveals when the assistant fabricates closures, hazards, or trail identities. Monthly sampling creates a recurring control that catches drift after corpus updates, model upgrades, or seasonal content changes before hikers rely on fiction. Documented miss rates also feed retraining, retrieval improvements, and release gates.

Sampling bot answers monthly and checking invented trail names against open web search results fails because web results are not the authoritative corpus; unofficial pages, outdated blogs, and search-indexed errors can affirm an invented closure that the approved park data would reject. Sampling bot answers monthly and scoring invented trail names for semantic similarity to corpus fails because similarity scores closeness in wording, not factual presence in the corpus; an invented name like Cedar Falls Loop can score highly against real Cedar and Falls terms while still naming no real approved trail. Sampling bot answers monthly and checking invented trail names against last year's archived corpus fails because the archive is stale; trails are added, renamed, reopened, or newly closed, so only the current approved corpus can confirm what exists and is closed now.

Exam caveat: Hallucination audits measure factual grounding against authoritative sources; they are related to but distinct from checking invented trail names against open web search results or scoring invented trail names for semantic similarity to corpus, which measure findability and wording closeness, not truth against the corpus. Checking invented trail names against open web search results fails to verify trail names against the authoritative corpus, so sampling against the approved corpus remains necessary. Also define what counts as a fail (invented name vs outdated but once-true closure), and do not substitute checking invented trail names against last year's archived corpus or against open web search results for an actual current corpus check.

Operational check: Each month, sample parks chatbot answers on closures and trail status; compare to the official GIS/corpus rather than checking invented trail names against open web search results or against last year's archived corpus; log hallucination incidents; block release on rising rates; and avoid relying on scoring invented trail names for semantic similarity to corpus without exact corpus verification.