311 operations want ongoing insight into an AI vendor after go-live. Which monitoring set best fits continuous vendor oversight?
Select an answer to reveal the explanation.
Short Explanation
Signing day isn’t the finish line. Watch uptime, hallucination incidents, and policy-violation tickets—or you’re flying without instruments.
Full Explanation
County IT operations that want ongoing insight into an AI vendor after go-live must monitor service health and policy adherence continuously. Uptime, hallucination incident rate, and policy-violation tickets best fit continuous vendor oversight because they measure availability of the municipal AI service, harmful or false outputs that disrupt county work, and breaches of acceptable-use or policy controls. After go-live, civic third-party AI governance depends on these operational signals—not ceremonial artifacts—to decide whether to remediate, escalate, or prepare for non-renewal.
A one-time kickoff selfie with the account manager fails by concept: a kickoff photo is ceremony, not continuous oversight metrics. Only annual review of the vendor’s holiday card design fails by concept: holiday-card aesthetics do not measure uptime, hallucinations, or policy violations. Stopping all metrics once the contract is signed fails by concept: signature begins the monitoring period; ending metrics at award blinds the county to post-go-live AI risk.
Exam caveat: metrics must be defined with thresholds and owners; collecting vanity dashboards without action criteria is insufficient. Hallucination incident rate depends on a reporting path—staff must know how to ticket suspected false outputs.
Operational check: instrument the AI vendor relationship with uptime SLIs, a hallucination/incident intake tag, and policy-violation ticket categories; review trends on a fixed cadence with the vendor manager; escalate sustained breaches; and feed metric outcomes into renewal scoring rather than relying on kickoff goodwill.