An AI operations team is deploying a large language model (LLM) for real-time customer service chatbot inference on an NVIDIA GPU. The team needs to maximize the system's inference throughput (tokens processed per second) while maintaining acceptable latency. Which of the following adjustments will most directly optimize GPU execution efficiency and throughput?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: deep learning models, especially massive LLMs, are incredibly hungry for memory bandwidth. When a single request comes in, the GPU spends most of its time just loading the model's weights from its high-bandwidth memory into its processing cores. That is a massive waste of processing power! If you process just one request at a time, your GPU is basically idling while waiting for memory. But check this out: if you group multiple requests together—called batching—you load those weights once and use them to process all those requests in parallel. By increasing your batch size (without running out of GPU memory, of course!), you keep those CUDA cores busy, boosting your throughput through the roof. Changing the CPU speed won't help because the bottleneck is on the GPU. A faster NIC only helps data get to the server, and queuing just rearranges the line. Trust me, batching is the key to unlocking GPU performance in production.
Full explanation below image
Full Explanation
In large language model (LLM) inference, the workload is highly memory-bandwidth bound. During the generation phase (autoregressive decoding), the model must read all its parameters from the GPU's High Bandwidth Memory (HBM) to the SRAM for every single token generated. Because the arithmetic intensity (computations per byte of memory transferred) is low for a single request, the GPU's tensor cores are underutilized.
By increasing the batch size (processing multiple requests simultaneously), the system can reuse the model parameters loaded into SRAM across all batched requests. This raises the arithmetic intensity of the operation, transitioning the workload from memory-bandwidth bound toward compute-bound. Consequently, batching significantly increases the total throughput (measured in tokens per second or requests per second) and improves GPU utilization efficiency. However, the batch size must be carefully tuned so that the activations and KV cache (key-value cache) do not exceed the GPU's physical memory capacity (VRAM), which would cause out-of-memory (OOM) errors.
Other options do not address the core bottleneck: - Upgrading CPU frequency (Option B) is ineffective because the inference calculations are executed on the GPU, leaving the host CPU mostly in a coordination role. - Upgrading the network interface card (Option C) enhances network transmission speeds but does not affect the computation time or resource utilization on the GPU. - Priority queuing (Option D) is a scheduling mechanism that manages user experience and service-level agreements (SLAs), but it does not modify the underlying performance characteristics or throughput of the GPU inference execution.