A 311 GenAI assistant has no monitoring phase KPIs after deployment. What should the auditor emphasize?

Establishing post-deployment monitoring KPIs as a required stage of the 311 GenAI assistant lifecycle works because go-live begins operations, it does not end assurance. AI-specific KPIs—answer groundedness, escalation rate, hallucination or refusal patterns, resident task completion, citation validity, and disparity across neighborhoods—detect drift and harm that infrastructure uptime alone will miss. Embedding monitoring in the methodology makes ownership, dashboards, thresholds, and response playbooks first-class lifecycle deliverables. The correct emphasis keeps the assistant under continuous quality supervision after deployment.

Treating go-live as the final lifecycle stage and monitoring only infrastructure uptime tickets fails because servers can be healthy while the assistant confidently misroutes residents or cites obsolete services. Tracking vendor foundation model benchmark scores as ongoing local performance monitoring KPIs fails because vendor benchmarks reflect generic model performance, not your deployed assistant's local accuracy, resident outcomes, or city-specific risks. Extending pre-deployment validation accuracy metrics into production as the ongoing monitoring framework fails because validation metrics are measured before launch and cannot detect post-deployment drift, real resident interactions, or emergent local harms.

Exam caveat: vanity metrics (raw chat volume) can look green while quality collapses—KPI design must prefer outcome and safety indicators over engagement alone. Operational check: publish a monitoring scorecard with owners and alert thresholds before production cutover, review it on a fixed cadence, and open incident tickets when AI quality KPIs breach limits even if uptime remains perfect.