Library GPU nodes need a controlled Kubernetes upgrade. What node-lifecycle practice should admins follow?
Select an answer to reveal the explanation.
Short Explanation
Cordon and drain first—like closing a lane and clearing cars before repaving. Upgrading under live scheduling or yanking nodes without a drain is how library GPU pods get stranded mid-job.
Full Explanation
Cordon then drain under change control is the standard node-lifecycle practice before upgrading library GPU nodes. Cordon marks the node unschedulable so new pods land elsewhere; drain evicts running pods respectfully (honoring PDBs and terminationGracePeriodSeconds) so GPU inference and training can reschedule onto healthy nodes before kubelet, drivers, or the node OS are upgraded. Documented change windows keep municipal library services predictable during the rollout.
Upgrading live nodes while pods keep scheduling onto them invites new work onto a half-upgraded kubelet or driver stack—classic stranding of GPU pods mid-job. Force-deleting every GPU node from the API with no drain step drops workloads immediately, skips PDB protection, and can leave orphaned device state and incomplete writes. Disabling kubelet logging to make maintenance look quieter only hides failure signals operators need when the civic upgrade misbehaves.
Exam caveat: drain is not instantaneous on GPU nodes—large models and long grace periods mean drains can take minutes; size the change window accordingly. Some DaemonSets (device plugins, GPU Operator) may need careful handling; cordon/drain order and version skew between control plane and workers still matter. Always upgrade control plane before workers per Kubernetes version skew policy unless your platform vendor documents otherwise.
Operational check: verify cordon (SchedulingDisabled), run a drain with a dry-run or graceful timeout, confirm library GPU pods are Ready on remaining nodes, then apply the upgrade. After uncordon, check nvidia-smi/device plugin health, pod GPU allocations, and that catalog or branch inference SLOs recovered.