An engineering team is scaling a deep learning model training run from a single system containing 8 GPUs to a distributed cluster of 32 GPUs spread across 4 server nodes. What are three critical technical requirements or considerations they must address to ensure effective multi-node performance? (Select three)
Select all correct answers, then click Submit.
Short Explanation
When you scale from one 8-GPU box to four 8-GPU nodes, you need low-latency, high-bandwidth interconnects like InfiniBand or RoCE, distributed frameworks such as DDP, DeepSpeed ZeRO, or FSDP, and careful tuning of global batch sizes and gradient accumulation steps. If you skip those, your GPUs spend more time waiting on the network than doing math, and you won't get perfectly linear speedups.
Full Explanation
Scaling deep learning training from a single node to a multi-node cluster introduces significant distributed systems challenges, primarily centered around synchronization and communication bottlenecks. The correct considerations are: High-speed interconnects: In distributed training, GPUs must constantly share and synchronize their calculated gradients. This synchronization is extremely bandwidth-intensive, and standard networking creates a severe bottleneck. Technologies like InfiniBand or RDMA over Converged Ethernet (RoCE) are required to provide the high throughput (for example, 200 Gbps to 400 Gbps per link) and sub-microsecond latency needed for efficient scaling. Distributed frameworks and sharding strategies: Specialized software libraries such as PyTorch Distributed Data Parallel (DDP), DeepSpeed with ZeRO technology, or Fully Sharded Data Parallel (FSDP) are necessary to orchestrate the distributed run. These frameworks handle the splitting of data (data parallelism), the splitting of model layers (pipeline parallelism), or the sharding of model states (ZeRO) across the cluster. Batch size and gradient accumulation: Because communicating across nodes is slower than communicating within a single node, developers must tune global batch sizes and gradient accumulation steps to balance computation time against inter-node communication latency. If the batch size per step is too small, the GPUs finish computing quickly and spend most of their time idle, waiting on network synchronization. Implementing gradient accumulation allows GPUs to run multiple forward/backward passes locally before performing an expensive global synchronization step, effectively amortizing communication overhead. The incorrect choices fail for the following reasons. The claim that a model footprint in each GPU's VRAM is automatically reduced without using sharding frameworks is wrong because simply distributing a workload across nodes does not reduce the VRAM required per GPU unless a specialized sharding framework such as ZeRO is explicitly implemented. The suggestion to rely on standard Gigabit Ethernet switches and CAT6 cabling because network latency has minimal impact is wrong because standard Ethernet and cabling are far too slow, will throttle GPU utilization, and lead to poor scaling efficiency. The expectation of perfectly linear speedups, where 32 GPUs complete the run in exactly one-quarter of the time of 8 GPUs, is wrong because distributed training scales sublinearly due to network communication overhead; 32 GPUs will not achieve a perfect 4x speedup over 8 GPUs.