How should a city profile agent reliability before trusting a housing chatbot in production?

Before trusting a housing chatbot in production, reliability profiling must include failure injection—specifically chaos tests that kill a retrieval pod mid-conversation and measure recovery. Housing intake agents depend on retrieval for policy text, eligibility rules, and case history; if retrieval dies mid-turn, the agent must degrade safely (clear resident messaging, retry with backoff, fallback corpora, or human handoff) rather than hallucinate eligibility. Chaos profiling validates timeouts, circuit breakers, session continuity, and whether Kubernetes restarts restore service within the city’s recovery objective.

Only testing when every dependency is guaranteed never to fail fails conceptually because that condition never holds in production municipal platforms; it measures an idealized path, not reliability. Skipping failure injection because residents never experience outages fails because residents do experience outages—and civic duty requires designing for them. Measuring only best-case latency on an idle weekend cluster fails because idle best-case hides contention, cold starts, and recovery behavior under fault.

Exam caveat: chaos tests must be scoped (namespace, blast radius, off-peak windows) and paired with observability so you can attribute resident-visible errors to the injected fault. Do not confuse “pod restarted” with “conversation recovered with correct housing guidance.” Also ensure PII in housing chats is not dumped into unrestricted crash logs.

Operational check: inject retrieval-pod kills during scripted housing dialogues, record recovery time, error rate, and whether answers remain policy-safe; require documented fallbacks and alert coverage before production trust is granted.