In a shared Kubernetes cluster running multiple concurrent deep learning training jobs and real-time inference microservices, which capability is essential to prevent resource starvation, isolate team workloads, and guarantee fair scheduling under heavy load?

Managing resource allocation in a multi-tenant Kubernetes cluster hosting both batch deep learning training and persistent real-time inference workloads is critical to avoid resource contention and starvation. The essential capability is namespaces coupled with resource quotas and priority classes. Namespaces provide virtual isolation, allowing administrators to group resources and users. Resource Quotas are applied to namespaces to enforce hard limits on compute resources such as CPU, memory, and GPU counts, ensuring no single project or user can monopolize the cluster's hardware. Priority Classes allow workloads to be prioritized. If the cluster runs out of capacity, Kubernetes can pre-empt or evict lower-priority pods, such as non-urgent training runs, to schedule higher-priority pods, such as user-facing inference APIs. The service mesh option for service-to-service communication manages traffic routing, security, and observability between microservices, but it does not address resource contention directly. The horizontal pod autoscaler option based on custom CPU/GPU metrics manages scaling the number of replicas up or down based on metrics, but without quotas, scaling out could actually exacerbate resource contention and crash the cluster. The integrated container image registry option for rapid image pulls manages image storage and delivery, which does not impact running resource constraints.