Evening 311 peaks need higher throughput across multiple agent models. Which Triton feature helps?
Select an answer to reveal the explanation.
Short Explanation
Rush hour at 311 is a convoy, not one car at a time. Triton dynamic batching across multiple agent models keeps evening peaks moving.
Full Explanation
Evening 311 peaks need higher throughput across multiple agent models. Triton’s dynamic batching helps by grouping concurrent inference requests efficiently so GPUs stay utilized while serving multiple municipal agent models during surge windows. Dynamic batching is a core inference-server capability for absorbing bursty civic traffic without forcing one-at-a-time cold execution.
Running each request on a cold-started empty GPU with no batching ever fails throughput by design—cold starts and empty utilization waste capacity at peak. Serializing all models onto one thread and rejecting concurrent 311 sessions fails resident service levels during the exact hours demand spikes. Turning off the inference server during peaks to “protect” GPUs fails inverted logic: peaks are when the server must be available and efficiently scheduled.
Exam caveat: dynamic batching improves throughput/latency tradeoffs within configured delay budgets; excessive batch wait can hurt tail latency. Tune max batch size and batch timeout against the city’s p95 SLO, and isolate critical models if noisy-neighbor effects appear.
Operational check: load-test an evening 311 profile with multiple models on Triton, compare throughput and p95 with dynamic batching on versus off, watch GPU utilization, and set alerts when batch queue delay threatens the civic latency budget.