A cloud-native Kubernetes cluster hosts a mixed set of AI workloads, including long-running, high-priority LLM training jobs and short, latency-sensitive inference services. To prevent resource conflicts and ensure that training jobs do not occupy nodes reserved for real-time inference, which Kubernetes scheduling mechanism should be configured?
Select an answer to reveal the explanation.
Short Explanation
You need to keep long-running training pods off the GPU nodes reserved for real-time inference, so use node affinity plus taints and tolerations to steer workloads to the right GPU pools. Taints block pods that don't belong, and tolerations let only the right workloads in. That keeps your inference latency clean and your training jobs where they belong.
Full Explanation
Managing a mixed-workload AI environment in Kubernetes requires robust resource scheduling to prevent resource contention, guarantee high availability for latency-sensitive tasks, and maintain high utilization for batch training jobs. The best mechanism to achieve this segregation and optimization is applying Node Affinity rules along with Node Taints and Tolerations to direct workloads to specific GPU node pools. Taints are applied to nodes, allowing a node to repel a set of pods. Tolerations are applied to pods, allowing, but not requiring, the pods to schedule on nodes with matching taints. For instance, an administrator can taint a pool of high-end GPU nodes, such as NVIDIA H100s, reserved for real-time inference with dedicated=inference:NoSchedule. Only inference pods configured with a matching toleration can run on those nodes, preventing long-running training jobs from accidentally scheduling on and monopolizing those resources. Node Affinity, on the other hand, is a set of rules that defines hard or soft preferences for scheduling pods on specific nodes based on labels, such as scheduling training jobs on nodes equipped with NVLink interconnects. Used together, these features allow granular, automated, and dynamic control over workload placement across heterogeneous GPU clusters. The global First-In, First-Out queue across the entire cluster namespace is wrong because it is a simplistic scheduling algorithm that processes jobs in the order they arrive. In a mixed environment, a FIFO queue would allow a massive batch training job to block urgent, real-time inference jobs, leading to unacceptable service delays. Manual static IP assignments and hostport mapping for every scheduled pod are wrong because they do not scale, bypass the Kubernetes scheduler's benefits, and introduce administrative overhead and single points of failure. Scaling up the request limits for GPU memory, limits.nvidia.com/gpu, on all workloads is wrong because it does not isolate workloads or prevent scheduling conflicts; instead, it can lead to scheduling failures because nodes will quickly appear fully allocated, leaving pods in a Pending state.