During deep learning model training on a multi-GPU cluster, you monitor the hardware metrics and observe that GPU utilization remains low (around 30%), while I/O wait times on the storage volumes are consistently high. Which infrastructure modification will resolve this bottleneck and increase GPU utilization?

In deep learning training, the storage subsystem must supply training data (such as images, video, or large text corpora) to the GPUs fast enough to keep their tensor cores fully occupied. If the storage media has high latency or low read throughput, the GPUs will sit idle, a state characterized by high I/O wait times and low GPU utilization. Replacing SATA-based HDDs or network shares with high-performance NVMe SSDs directly addresses this because NVMe devices communicate over the high-speed PCIe bus, offering gigabytes-per-second throughput and parallel queues that drastically reduce I/O bottlenecks compared to legacy SATA drives or standard mechanical HDDs.

Why the other approaches fail to solve the bottleneck: - Decreasing the batch size in the training configuration actually increases the frequency of I/O requests. Each epoch will require more steps, which increases the overhead of the storage request pipeline and can worsen the CPU/storage bottleneck. - Implementing additional real-time CPU-bound data augmentation techniques increases the CPU workload per batch, which could further delay data transfer to the GPU if the CPU is already struggling to keep up. - Scaling out the cluster by adding more GPUs increases the overall demand on the storage subsystem. If the storage cannot feed the existing GPUs, adding more GPUs will only decrease individual GPU utilization further, resulting in higher hardware costs with no performance gain.

To maximize throughput, high-performance AI architectures often combine NVMe storage with technologies like GPUDirect Storage, which bypasses the CPU entirely and moves data directly from NVMe storage to GPU memory.