County health measures NCCL east-west bandwidth well below the fabric design target. What should operators do next?
Select an answer to reveal the explanation.
Short Explanation
Low NCCL E/W bandwidth is a fabric clue, not a rounding error—dig into congestion and cabling before county health pretends the design target was optional.
Full Explanation
When county health measures NCCL east-west bandwidth well below the fabric design target, operators must investigate fabric congestion, cabling, and path issues that can suppress collective throughput. Low NCCL E/W numbers are a Domain 4 signal that the AI fabric is not delivering the bandwidth training collectives require, even if individual GPUs still enumerate. Ignoring the shortfall because GPUs still appear in nvidia-smi confuses device visibility with fabric performance; nvidia-smi does not prove all-reduce bandwidth across rails. Raising the SLO on paper until the measured number looks fine rewrites the acceptance criteria instead of fixing the network and hides a real capacity defect from municipal stakeholders. Disabling NCCL permanently and training with host-loopback only abandons the communication path multi-node AI depends on and cannot validate or remediate the fabric. Exam caveat: root causes can be layered—bad DAC/AOC, wrong rail mapping, PFC/ECN misconfiguration, oversubscribed uplinks, or a single noisy neighbor—so investigate systematically rather than assuming one failure mode. Operational check: for county health, capture the failing NCCL topology and bandwidth report, walk the implicated links and switch counters for errors or congestion, correct pathing or QoS, then re-run NCCL until east-west results meet the documented design target before returning capacity to training.