After maintenance, Run:ai still schedules to a cordoned civic GPU node. What should admins do?
Select an answer to reveal the explanation.
Short Explanation
If Kubernetes says cordoned, Run:ai should agree—reconcile the state so jobs skip that civic node. Forcing pods onto it or blindly uncordoning sick hosts invites failures. Nuking the whole cluster is overkill.
Full Explanation
After maintenance, Kubernetes may cordon a civic GPU node so pods are not scheduled there, but Run:ai can drift if its scheduling view is stale. Admins must reconcile Run:ai scheduling state with the Kubernetes node condition so municipal jobs skip the cordoned node until it is healthy and uncordoned. Reconciliation prevents allocations onto nodes intentionally taken out of service. Ignoring Kubernetes cordon state and forcing pods onto the node defeats maintenance safety. Wiping the entire cluster including healthy nodes is disproportionate and causes a wider outage. Uncordoning every failed node without checking health returns broken capacity to the schedule. Exam caveat: cordon stops new pods but does not always drain running ones—confirm drain/maintenance procedures separately from Run:ai reconciliation. Operational check: cordon a test GPU node, confirm Run:ai no longer places new municipal jobs there, fix the node, uncordon only after health checks, and verify scheduling resumes solely on ready nodes. After maintenance, compare kubectl get nodes conditions with Run:ai node/status views and clear any cached allocatable flags on cordoned hosts. Civic platform admins should treat scheduler reconciliation as part of the change close-out checklist so the next training wave does not land on a node still marked unschedulable in Kubernetes only.