A water utility depends on a vendor-hosted model that updates without notice. Which monitoring practice best detects silent behavior change?

Deploying canary prompts to detect silent model-behavior changes after vendor updates is the monitoring practice that fits a water utility dependent on a vendor-hosted model that may change without notice. The correct practice works because fixed canary inputs with expected response properties reveal drift in accuracy, refusal behavior, toxicity, or domain advice as soon as a silent update lands—before operators discover it through bad field decisions. Tracking input embedding drift to detect silent upstream data pipeline shifts post-update fails to detect vendor-model change because input representations can remain stable while a silent update still alters model outputs. Deploying canary prompts to benchmark vendor inference speed after capacity upgrades fails as a performance check because inference-speed benchmarks do not reveal silent changes in model outputs. Scoring golden answer overlap to detect silent evaluation harness version skew post-update fails by auditing the measurement harness rather than live vendor-model behavior. Exam caveat: canaries must cover utility-critical scenarios (safety advice, regulatory phrasing, numeric thresholds) and be versioned so teams know which expectations apply after intentional prompt changes. Operational check: maintain a canary prompt suite with pass/fail criteria, run it on a schedule and on suspected update events, alert water-utility AI owners on failures, and pause dependent automations until drift is understood and accepted or mitigated.