Inference-heavy municipal agent services need Kubernetes HPA. Which signals should drive scaling?
Select an answer to reveal the explanation.
Short Explanation
You’ll get a better fit for inference-heavy agents by watching GPU and CPU signals, since accelerator-bound work can keep GPUs busy while CPU use looks moderate. Help-desk email counts and hallway thermostat readings don’t track inference demand, and a fixed replica count can’t adapt when demand changes.
Full Explanation
Inference-heavy municipal agent services should drive Kubernetes HPA with GPU metrics as well as CPU metrics. Agent pods can spend much of their time on accelerator-bound token generation, so CPU-only signals may under-react when GPUs are saturated but CPU use remains moderate; queues for services such as 311 or permitting can then grow. GPU and CPU signals let scaling respond to the resources the inference workload actually uses. Scaling only on unread IT help-desk emails fails because human ticket volume does not measure demand on the inference cluster. Office thermostat readings in the data center hallway are an unrelated facility measurement, not a signal of pod workload or capacity. A fixed replica count that never changes after day one cannot respond to demand that varies by time or events such as storms and tax deadlines, risking wasted budget at quiet times or inadequate capacity at peaks. Exam caveat: GPU metrics must be reliably exposed to HPA through the device plugin, DCGM, or a custom metrics adapter; noisy short spikes can cause scaling thrash, and GPU expansion still needs cost controls. Practical check: export GPU utilization into the metrics pipeline, configure HPA to use GPU and CPU, load-test an inference spike, and verify replicas rise before resident-facing SLOs are breached.