After a tariff change, a water utility’s billing FAQ retrieval starts returning outdated policy chunks. What should AI ops monitor?

After a tariff change, AI ops should monitor retrieval relevance performance against the updated water-utility billing knowledge base so the FAQ assistant surfaces current policy chunks. The practice works because tariff updates invalidate prior embeddings and document versions; relevance metrics, labeled query sets, and citation checks reveal whether retrieval still ranks obsolete rate language above the new schedule. Monitoring relevance is the operational bridge between billing policy publication and trustworthy FAQ answers.

Monitoring retrieval recall performance against the prior billing knowledge base snapshot fails because it measures how well retrieval finds the old pre-tariff documents; strong recall on that snapshot can persist while the system still ranks obsolete rate language above the new schedule. Measuring response grounding fidelity over historical billing FAQ passages fails because fidelity only shows the answer matches the retrieved old passages; a response can be fully grounded in historical billing text and still quote pre-tariff-change rates. Monitoring chunk index refresh latency for outage notification document collections fails because it tracks freshness for a different collection; fast outage-document indexing says nothing about whether billing FAQ retrieval still surfaces outdated policy chunks.

Exam caveat: keyword hit rate alone can look fine while semantic ranking still prefers longer outdated PDFs; include graded relevance and “must-cite new tariff ID” checks. Operational check: rebuild or refresh the billing index after tariff publish, run a fixed query pack covering rate classes residents ask about, alert on relevance regressions, and verify answers cite the new effective-date documents before closing the change ticket.