County health sees links up but NCCL performance far below design after a re-cable. What cabling fault should operators suspect?
Select an answer to reveal the explanation.
Short Explanation
You can see links come up even when the rails are crossed, so check that the cabling matches the topology NCCL expects. Topology matters for collective performance, not just link status.
Full Explanation
When links are up but NCCL performance falls far below design after a re-cable, suspect crossed rails that establish link while breaking the expected topology. A live link confirms connectivity, not correct rail alignment; crossed connections can disrupt the paths and peer mappings that collective communication relies on for bandwidth. The claim that correctly installed cables are the issue, or that cabling has no meaningful effect on collective bandwidth, misses that the physical topology is essential to performance. DNS misconfiguration on a management laptop affects name resolution on the management plane, not GPU-fabric collective throughput. Keyboard layout settings on the BMC console affect text input, not interconnect topology. Exam caveat: link state alone does not prove the cabling matches the intended topology, so distinguish physical connectivity from correct rail mapping. As a practical check, compare the as-built port connections with the county health rail design, correct any crossed ends, then rerun a multi-node NCCL bandwidth test and verify that performance returns to the design baseline.