A engineering team is training a large language model across an NVIDIA DGX cluster using distributed data parallel training. During monitoring, they notice poor scaling efficiency: as they add more nodes, the GPUs remain underutilized, and training throughput does not increase linearly. Which of the following issues are the most likely causes of this distributed training bottleneck? (Select two)
Select all correct answers, then click Submit.
Short Explanation and Infographic
Check this out: when you scale out to multiple DGX nodes, your GPUs have to talk to each other constantly to share gradients during the backward pass. Think of it like a team of developers trying to write a single app—if the internet connection between their houses is a dial-up modem, they are going to spend all day waiting to sync their code instead of actually typing. That's your node interconnect bandwidth. If you're using slow Ethernet instead of InfiniBand, your GPUs sit idle waiting for data. The other half of this equation is NCCL (NVIDIA Collective Communications Library). NCCL is the software secret sauce that makes multi-GPU communication lightning fast. If you misconfigure it or bypass it, you lose all that optimization, and your cluster performance tanks. Keep an eye on both the hardware link and the communication library!
Full explanation below image
Full Explanation
Distributed deep learning, particularly with large models, relies heavily on rapid communication between GPUs across different nodes to synchronize gradients (e.g., during All-Reduce operations). The two primary components required for high-efficiency multi-node scaling are:
1. Physical Interconnect Bandwidth: In multi-node setups, nodes must exchange massive amounts of model parameter data. If the physical network connecting the nodes is slow (for instance, using standard 10GbE instead of dedicated high-speed NVIDIA Quantum InfiniBand or RoCE at 100/200/400 Gbps), the network becomes a severe bottleneck. The GPUs will spend a significant portion of their time idle, waiting for network transfers to complete (known as being communication-bound), leading to low GPU utilization. 2. Collective Communication Optimization (NCCL): NVIDIA Collective Communications Library (NCCL) provides multi-GPU and multi-node collective communication primitives (such as All-Gather, All-Reduce, Broadcast, and Reduce) that are highly optimized for NVIDIA GPUs. If NCCL is improperly configured, disabled, or if the environment variables (like NCCL_DEBUG, NCCL_IB_DISABLE, or interface selections) are misconfigured, the system may fallback to slower communication paths (like host TCP/IP), which severely increases latency and degrades distributed training scaling.
Let's review the incorrect options: Option A: CUDA cores cannot be individually configured or "misconfigured" in terms of physical layouts; their clock speed and performance are managed automatically by the driver and runtime. Option C: A lack of GPU memory (VRAM) would cause an out-of-memory (OOM) crash during the allocation phase, not poor scaling efficiency or GPU underutilization during execution. Option E: While incorrect model parallelism can cause performance issues, standard distributed data parallel (DDP) is typically used for cluster scaling, and incorrect parallelism implementation usually results in runtime crashes or incorrect mathematical outputs rather than purely underutilizing communication resources on multi-node runs.