Production Monitoring and Reliability

NCP-GENL · 14 questions

• City AI ops is standing up reliability visibility for the permitting LLM. Which dashboard metric set best matches production monitoring needs?
• Public-health wants the clinic chatbot board to reflect answer quality, not only whether the service is up. What should they add beside uptime?
• Transit leadership cares whether the rider assistant stays available when morning peak load hits. Which reliability metric should the SLO board treat as first-class?
• Housing chat starts returning HTTP 500s. Structured logs show vector database timeouts lining up with the spike. What is the right monitoring response?
• After a tokenizer update, the clerk bot suddenly returns many empty responses. How should ops treat that signal?
• An overnight deploy made the emergency alert rewriter behave oddly. Which logging practice best supports root-cause diagnosis?
• Each week the city wants proof the live ordinance QA LLM has not silently regressed versus last month. What practice fits?
• Housing chat win-rate versus the prior version drops right after a LoRA swap. What should production reliability practice trigger?
• Public works wants nightly evidence on a candidate LLM without exposing residents to it yet. Which approach fits continuous version benchmarking?
• The city wants the 311 intent head refreshed on a schedule with traceable artifacts. Which production practice fits?
• Parks chat keeps breaching latency SLOs. Ops wants production automation before a full model retrain. What fits?
• Courts require that any production clerk LLM can be reproduced exactly from registry state. Which versioning discipline is required?
• Residents ask whether the permitting assistant is trustworthy and currently working. Which transparency practice supports uptime trust?
• Emergency information LLM answers must continue if one city GPU pool fails. Which reliability architecture fits?