Under the city’s latency SLO, unoptimized Hugging Face serving fails for the public agent. What should the team select?
Select an answer to reveal the explanation.
Short Explanation
If the stopwatch fails the city’s SLO, change the engine—not the rulebook. TensorRT-LLM plus Triton is the selection when unoptimized HF serving can’t keep up.
Full Explanation
Under the city’s latency SLO, if unoptimized Hugging Face serving fails for the public agent, the team should move to TensorRT-LLM with Triton Inference Server. That pairing is the NVIDIA path for production-grade optimized inference: TensorRT-LLM accelerates the model runtime; Triton provides scalable serving, batching, and operational controls needed for civic SLOs that informal HF serving often misses under concurrent resident load.
Keeping unoptimized HF serving and lowering the SLO until failures disappear on paper fails governance—it rewrites the stopwatch instead of fixing performance. Asking residents to retry until responses eventually succeed fails service design and equity: retries shift outage cost onto the public. Disabling SLOs entirely so latency cannot be marked as failing fails accountability; unmarked failure is still failure for 311 callers.
Exam caveat: selecting TensorRT-LLM + Triton addresses serving performance; it does not automatically fix slow tools, oversized contexts, or missing caches. Bring traces to show the bottleneck is model inference before declaring victory solely from a stack swap.
Operational check: benchmark the same public-agent workload on unoptimized HF versus TensorRT-LLM on Triton, verify p95 meets the written civic SLO, run quality and safety regressions, then cut traffic gradually with error-budget burn monitoring.