Water leak-detection inference is getting a new model. How should the Kubernetes rollout start?
Select an answer to reveal the explanation.
Short Explanation
Canary means one GPU node tries the new leak model while the rest keep the old map—like pilot hydrants before citywide change. Big-bang cutovers and killing the Service first remove your civic safety net.
Full Explanation
Canary the new water leak-detection model on one GPU node (or a small replica subset) before wider civic rollout. Keep the old Deployment/ReplicaSet serving most traffic, shift a controlled slice to the new version, compare accuracy, latency, and false-alarm rates against hydrant/sensor ground truth, then promote. That preserves a rollback path while proving the model in production-like conditions.
Replacing every production replica worldwide in one simultaneous update is a big-bang cutover—any defect becomes a citywide outage. Deleting the old Service first so no traffic can fall back removes the safety net during testing. Skipping staging and only monitoring after a full weekend cutover delays detection until the blast radius is already large.
Exam caveat: canary success metrics must be defined up front (leak precision/recall, p95 latency, GPU OOM rate). Use separate labels/selectors or a service mesh/Ingress weight—not manual pod IP edits. Ensure both versions can run concurrently without claiming exclusive node resources the canary still needs.
Operational check: run the single-node canary under real sensor feed, compare dashboards to the baseline fleet, document go/no-go, then roll forward with a controlled Deployment strategy (maxUnavailable/maxSurge) while watching civic false-alarm tickets.