An AI operations team notices that during peak hours, the latency of their LLM inference service spikes due to a sudden surge in user requests. Which Kubernetes component should they configure to automatically spin up additional replicas of their inference pods to handle the increased load based on CPU, memory, or custom metrics?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: in the real world, AI inference workloads are anything but steady. One minute your LLM is sitting idle, and the next minute your boss walks in and announces a product launch that sends request traffic through the roof. If you don't scale, your response times will crawl and your users will start complaining. To fix this, you want to use the Horizontal Pod Autoscaler, or HPA. Think of the HPA like a smart thermostat for your deployment. You set the target—say, keep CPU usage at 70% or set a custom metric for request volume—and the HPA monitors your pods. When things get hot and utilization climbs past that threshold, HPA automatically tells Kubernetes to spin up more pods to share the load. When the rush is over, it scales them back down so you aren't burning cash on idle GPUs. Now, let's talk about the other components so you don't get fooled on the exam. The kube-scheduler is the matchmaker of the cluster—it looks at a newly created pod and decides which worker node has the room to run it, but it doesn't create new pods or decide to scale. The kubelet is the agent running on each individual node that actually starts and stops containers to make sure they're healthy; it doesn't look at cluster-wide scaling metrics. And while Triton is awesome at serving models, it doesn't handle the infrastructure scaling of Kubernetes. Remember: HPA is your go-to for adjusting pod count on the fly. Let's keep rolling!
Full explanation below image
Full Explanation
The Horizontal Pod Autoscaler (HPA) is a standard Kubernetes controller that automatically scales the number of Pods in a replication controller, deployment, replica set, or stateful set. HPA operates by periodically querying resource utilization metrics (such as CPU and memory) from the resource metrics API, or custom/external metrics (like request rate or queue depth) from the custom metrics API. Based on these observed values, the HPA calculates the required number of replicas to maintain the user-defined target utilization and updates the deployment's configuration accordingly. This mechanism is critical for AI inference deployments, where traffic can be highly volatile and latency-sensitive.
Analyzing the incorrect options: - kube-scheduler (Option A) is responsible for watching for newly created Pods that have no node assigned, and selecting the optimal node for them to run on based on resource availability, constraints, and affinity rules. It does not scale the number of pods. - kubelet (Option C) is the primary node agent that runs on every worker node in a Kubernetes cluster. It receives PodSpecs (primarily from the API server) and ensures that the containers described in those specs are running and healthy. It has no visibility or control over horizontal scaling across the cluster. - NVIDIA Triton Inference Server (Option D) is an inference serving engine designed to maximize GPU utilization. While Triton optimizes model execution (including dynamic batching and concurrent model execution), it does not manage the orchestration-level scaling of container replicas in Kubernetes.
For production AI deployments, HPA is often paired with the Kubernetes Cluster Autoscaler to ensure that if the cluster runs out of physical nodes to host the scaled-up pods, new virtual or physical nodes are automatically provisioned.