County health single-node NCCL shows asymmetric bandwidth across GPU pairs on an NVLink Switch system. What should operators suspect?
Select an answer to reveal the explanation.
Short Explanation
Asymmetric NCCL inside one box is a red flag for NVLink Switch or link trouble—don't blame DNS on the login node.
Full Explanation
Asymmetric bandwidth across GPU pairs during county health single-node NCCL on an NVLink Switch system should lead operators to suspect possible NVLink Switch or intra-node link issues and to perform topology and switch-path diagnosis. Healthy NVLink Switch fabrics present balanced pair performance for a given topology; large asymmetry is a localization clue inside the node rather than a login-node DNS or HPL-parameter problem.
Treating asymmetry as normal behavior that never needs investigation fails by concept—material imbalance is a defect signal on these systems and should gate acceptance for flu-season inference hosts. Blaming only DNS misconfiguration on the login node fails because login DNS does not govern GPU-pair NCCL bandwidth inside the server. Claiming HPL settings alone explain NCCL asymmetry fails because HPL parameters do not create pairwise NVLink bandwidth imbalance in NCCL tests; the measurements and failure domains differ.
Exam caveat: Domain 4 NCCL interpretation links single-node asymmetry to NVLink Switch and intra-node link diagnosis, not to unrelated control-plane network services or Linpack knobs.
Operational check: capture per-pair NCCL bandwidth matrices, dump NVLink topology, inspect switch-path health and cable or seating conditions, remediate the defective path, and rerun until pair bandwidth sits within peer-expected symmetry bands before returning the health node to service.