An AI engineering team notes that their GPU-accelerated model training is running far below expected speed. Profiling reveals a PCIe bottleneck: the GPU is frequently idling while waiting for the host CPU to transfer new data blocks. Which of the following configuration changes would most effectively reduce the frequency of CPU-to-GPU data transfers and alleviate this communication bottleneck?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: if you have a powerful GPU but your system is bottlenecked by the PCIe bus, it's like having a Ferrari but being stuck in gridlock on a narrow bridge. The GPU finishes its calculations in milliseconds and then has to sit there twiddling its thumbs waiting for the CPU to send the next tiny packet of data. That's a classic data transfer bottleneck. How do we fix it? We increase the batch size! Instead of sending small packages of data back and forth a million times, we pack a much larger batch of data and send it at once. This utilizes the PCIe bandwidth much more efficiently and keeps the GPU busy computing instead of waiting. Just make sure you don't exceed your VRAM limit! Got it? Sweet. Let's keep rolling.
Full explanation below image
Full Explanation
In GPU-accelerated deep learning, the data pipeline involves loading data from disk to host system memory (RAM), preprocessing it on the CPU, and then transferring it to the device memory (VRAM) via the PCIe bus. When the batch size is small, the overhead of initiating PCIe transfers dominates, and the GPU finishes processing the batch faster than the CPU can feed it the next one, leading to low GPU utilization and slow training.
By increasing the batch size, you package more samples into a single tensor transfer. While the transfer of a larger batch takes slightly longer, the relative overhead is significantly reduced, and the GPU works on a larger chunk of compute, which keeps it active longer. This balances the communication-to-computation ratio, helping to bypass the PCIe interface bottleneck.
Let's analyze the incorrect options: Option A (pre-loading the entire dataset) is ideal in theory, but in practice, real-world deep learning datasets (such as ImageNet or large text corpora) are hundreds of gigabytes or terabytes, far exceeding the physical VRAM capacity of even high-end enterprise GPUs (like the H100 or A100). Option B (faster CPU clock speed) does not address the main issue, which is the bandwidth limits and transfer overhead of the PCIe bus. The latency of setting up driver calls remains the dominant bottleneck. Option D (using multiple GPUs without modifications) can actually worsen communication bottlenecks, as it introduces additional multi-GPU synchronization overhead (all-reduce) over the PCIe or NVLink channels without resolving the host-to-device transfer efficiency.