A shared multi-tenant NVIDIA GPU cluster is running several machine learning workloads. Recently, non-critical background batch jobs have been consuming disproportionate GPU memory and compute cycles, preventing high-priority, real-time inference services from executing reliably. How should the infrastructure administrator configure the cluster to guarantee resource availability and priority for the critical services?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: if you let background jobs run wild on a shared GPU cluster, they'll hog the hardware and starve your critical apps. You need Kubernetes to enforce GPU requests, limits, and quotas so high-priority services get guaranteed capacity and low-priority jobs get throttled or preempted.
Full Explanation
In a multi-tenant or shared GPU cluster, resource starvation is a common issue when workloads are not actively governed. To prevent low-priority batch jobs from consuming excessive GPU resources and degrading high-priority, latency-sensitive services, administrators must implement container orchestration resource controls. The correct approach is to define GPU resource requests, limits, and quotas within Kubernetes resource management to enforce priority scheduling and allocation. Kubernetes, when integrated with the NVIDIA Device Plugin, allows administrators to schedule GPUs just like CPU and memory resources.
By defining: - Resource Requests: The minimum amount of GPU resources a container needs to be scheduled. - Resource Limits: The maximum amount of GPU resources a container is allowed to consume. - ResourceQuotas: Cluster-level limits per namespace to restrict total GPU consumption. - PriorityClasses: Kubernetes objects that define the scheduling queue order, allowing the orchestrator to preempt (evict) lower-priority pods to free up GPU capacity for incoming high-priority pods.
The approach that offloads all low-priority workloads to CPU-only nodes, disabling GPU execution for those tasks, is inefficient and heavy-handed. Low-priority jobs may still benefit from GPU acceleration; they simply need to be throttled or preempted when high-priority tasks require resources. The approach that upgrades all physical GPUs in the cluster to models with larger memory capacities is expensive and does not solve the underlying resource-scheduling problem; larger GPUs will still be subject to starvation if there are no scheduler constraints. The approach that implements model quantization and pruning techniques on the high-priority models to reduce their resource footprints reduces the footprint of the high-priority model but does not prevent a low-priority job from scaling up and consuming the remaining cluster resources.