Transit finds mixed MIG and non-MIG GPU nodes confuse the scheduler. What should admins do?
Select an answer to reveal the explanation.
Short Explanation
Pick one MIG story per pool—like painting consistent stall sizes in each garage level—so the scheduler stops guessing. Nightly random layouts and contradictory labels just create civic scheduling chaos.
Full Explanation
Standardizing MIG profiles per municipal node pool makes GPU shapes predictable to the Kubernetes scheduler and device plugin. Transit ops saw mixed MIG and non-MIG nodes advertise inconsistent resources; pods that request a specific MIG instance type fail or pend when landed on full-GPU-only nodes, and vice versa. One profile story per pool—backed by consistent labels, taints/tolerations, and RuntimeClass or resource names—lets the scheduler place work reliably.
Randomizing MIG layouts nightly so no team can learn the topology guarantees scheduling thrash and broken Deployments after every reset. Advertising contradictory MIG and full-GPU labels on every node lies to the scheduler: matching pods may bind to nodes that cannot satisfy the real device inventory. Removing all GPU labels and letting pods guess device availability forces trial-and-error binding and Pending pods across the civic fleet.
Exam caveat: standardization is operational, not automatic—node pools, ClusterPolicies (GPU Operator), and IaC must encode the chosen geometry. When migrating a pool between MIG and full-GPU mode, drain first and update workload requests together. Document the resource API names pods must request for each pool.
Operational check: inventory nvidia.com/mig-* versus full GPU resources per pool, confirm labels match reality, and run a canary pod with each expected request shape. Alert on Pending GPU pods older than a short threshold so topology drift is caught before transit jobs miss their windows.