Run, Monitor, and Maintain

NCP-AAI · 10 questions

• Ops must watch the live 311 agent’s health beyond “is the pod up?” Which dashboard definition best fits?
• A civic AI dashboard already shows uptime. What additional reliability metrics should ops track for the agent itself?
• “Parcel not found” errors spike after a GIS change. How should ops find the root cause?
• A permit answer fails in production across orchestrator, NIM, and vector DB. What diagnostic approach helps most?
• Before a new 311 agent build takes full traffic, ops wants proof it is not worse than last week. What continuous benchmarking practice fits?
• English scores on the city’s 311 agent rose, but Spanish task scores dropped. What continuous benchmarking gate should block promotion?
• City MLOps stores prompt revisions for the permit agent. When should a new prompt version auto-promote to production?
• An embedding refresh makes ordinance retrieval worse for the municipal agent. What production practice limits damage?
• 311 tools start failing but the city still wants resident trust. What uptime and transparency practice fits?
• The live municipal agent invents a fake fee rule and residents notice. What ops practice maintains trust during the incident?