A public-works pavement-scan training job shows poor all-reduce times. Engineers review GPU-to-GPU communication paths in the pod. What should they evaluate first?
Select an answer to reveal the explanation.
Short Explanation
Slow all-reduce on a pavement-scan job is often a path problem: are the GPUs chatting on the fast local interconnect, or bouncing across a congested fabric like delivery vans stuck on surface streets? Check NVLink/NVSwitch locality first, then fabric hops and congestion for the multi-node pieces.
Full Explanation
When a public-works pavement-scan training job shows poor all-reduce times, engineers should first evaluate whether collectives use local NVLink/NVSwitch paths versus fabric hops, and whether fabric congestion is hurting multi-node traffic. All-reduce performance is dominated by the slowest segment of the communication path: on-node pairs should ride NVLink/NVSwitch, while cross-node pairs depend on clean Spectrum or InfiniBand east-west. Mapping those domains quickly separates “local interconnect sick” from “fabric congested” for civic computer-vision training.
NVSwitch caching improves locality for repeated operations but does not address multi-node latency, fabric congestion, or path locality problems that dominate all-reduce times. Incorrect rank pinning increases unnecessary hops but is secondary to the primary bottleneck of fabric congestion and cross-rail communication paths. Batching tensors reduces per-operation overhead but does not address the root cause of poor all-reduce latency, typically insufficient fabric bandwidth or poor path locality. Each is a plausible-sounding optimization, but none of them addresses the actual symptom: a congested or poorly placed communication path for cross-node traffic.
Exam caveat: start with GPU communication domains (NVLink/NVSwitch versus fabric) before chasing secondary tuning knobs like caching, rank placement, or batching. Operational check: capture NCCL/profiler evidence of intra-node versus inter-node times, inspect NVLink health and fabric link/ECN/PFC or IB congestion counters during the pavement-scan job, and only then widen scope; do not spend the first triage cycle re-tuning caching, rank placement, or batch size before the path data is in hand.