A civic AI dashboard already shows uptime. What additional reliability metrics should ops track for the agent itself?
Select an answer to reveal the explanation.
Short Explanation
Uptime just means the lights are on—not that answers are faithful. Add faithfulness and refusal-correctness next to classic SLIs for civic AI reliability.
Full Explanation
A civic AI dashboard that already shows uptime still needs agent-specific reliability metrics. Ops should track faithfulness and refusal-correctness alongside classic uptime SLIs. Faithfulness asks whether answers stick to retrieved municipal sources; refusal-correctness asks whether the agent correctly declines out-of-scope or unsafe asks. Together they reveal quality and safety failures that “pod up” never shows.
Replacing all metrics with a single weekly anecdote from one clerk fails as measurement—anecdotes lack coverage and statistical power. Measuring only how many emoji the agent uses per reply fails by tracking style trivia instead of correctness. Dropping refusal metrics so unsafe answers look like successful completions fails ethically and analytically: it converts safety failures into false success.
Exam caveat: faithfulness and refusal-correctness require labeled eval sets and consistent judges (human or automated rubrics). Without shared definitions, teams will game scores. Uptime remains necessary; it is simply insufficient alone for municipal AI.
Operational check: add faithfulness and refusal panels next to uptime, score each release on a fixed gold set, alert on regressions, and sample production traces monthly to calibrate whether online behavior matches offline metric trends.