A 311 call-center Triton config sets max batch size to an enormous number for speed, and a storm-day spike exhausts GPU memory. What is max batch size actually for?
Select an answer to reveal the explanation.
Short Explanation
An enormous max batch size is not a free speed switch. It is a memory-safe ceiling on how many requests share one pass. Storm-day spikes prove bigger is not always better. It is not NCCL world size or a video-frame cache.
Full Explanation
Max batch size caps how many requests share one forward pass so the GPU does not run out of memory. Raising it without a memory budget is not free speed. It is not NCCL world size, and it is not a video-frame cache. Treat the cap as a safety limit, then use dynamic batching under that ceiling when you want more throughput.