After a parks-and-rec field-reservation helper raised Triton’s max queue delay so batches could grow, residents say replies feel stickier. What does that queue delay control?
Select an answer to reveal the explanation.
Short Explanation
A longer max queue delay lets Triton wait so batches grow, and each resident waits a bit more. That is the latency-versus-throughput knob. It is not an NCCL window, a CUDA-stream pin, or a multimodal token budget.
Full Explanation
Triton’s max queue delay is the associate latency-versus-throughput trade: the server may wait a little longer so more requests share a batch, which can raise GPU throughput while adding wait time for each user. That is a serving knob, not a training collective. NCCL all-reduce windows and datacenter CUDA-stream pinning are Professional or ops topics, not this exam item. A multimodal token budget does not explain why a text reservation bot suddenly feels sticky.