A county KPI pack for AI tracks uptime but not model confidence calibration. What should the auditor recommend?
Select an answer to reveal the explanation.
Short Explanation
Uptime says the lights are on; calibration says the AI knows when it is guessing. Put AI-specific quality signals in the KPI pack. Availability alone is a thin report card.
Full Explanation
County AI supervision KPIs that track only uptime miss whether models are well calibrated—whether expressed confidence matches actual correctness on municipal tasks. A miscalibrated system can be continuously available yet systematically overconfident or underconfident on county decisions that affect residents. The auditor should recommend including AI-specific quality signals such as confidence calibration and related quality metrics in supervision packs, not only infrastructure availability.
Tracking infrastructure latency and error rates as proxies for model calibration fails because server health is not decision quality; a system can be fast and error-free yet still miscalibrated on county decisions. Reporting calibration metrics only internally to operations, without including them in the county KPI pack itself, keeps county leadership and the board blind to AI quality risk even though the data exists somewhere in the building. Retaining an uptime-focused KPI pack and reviewing calibration only at annual procurement renewal fails because AI quality risk needs continuous oversight, not a once-a-year check tied to buying decisions.
Exam caveat: AI supervision metrics are broader than classic IT SLAs—calibration, error types, drift, and impact indicators matter on the exam and in practice. Operational check: add calibration or confidence-reliability charts to the county AI KPI pack with named owners and thresholds, and present the board both availability and AI quality signals in the same reporting cycle.