When profiling a GPU-backed 311 agent fleet, why separate GPU utilization and queue depth from application RPS?

When profiling a GPU-backed 311 agent fleet, GPU utilization and queue depth must be separated from application requests-per-second because they measure different layers. RPS reflects admission at the request edge; GPU utilization and inference queue depth reflect whether accelerators can absorb that work. A quiet RPS chart can coexist with saturated GPUs, growing queues, and rising tail latency—exactly when residents wait on 311 chat during storms or billing cycles. Separating compute-layer metrics from request-layer metrics lets operators right-size GPU pools, tune batching, and set HPA signals that actually track inference health.

Claiming RPS alone always proves GPUs are healthy fails conceptually because RPS can be limited by upstream throttles, cached replies, or client retries while GPUs are already maxed on fewer heavy prompts. Restricting city dashboards to a single number forever fails because multi-layer systems need multi-layer signals; collapsing everything into one KPI hides the bottleneck class. Treating queue depth as irrelevant to agentic inference fails because queue depth is often the earliest leading indicator of overload for batched or token-heavy agent workloads.

Exam caveat: also correlate token throughput and batch size; high GPU util with empty queues can still mean inefficient kernels or oversized models, while low util with deep queues can mean scheduling or I/O stalls. Civic SLOs should bind resident-facing latency to both layers.

Operational check: dashboard RPS, GPU util, inference queue depth, and p95/p99 together; alert when queues grow even if RPS looks “fine,” and validate that scaling actions reduce queue depth without blowing GPU spend.