A housing authority multi-GPU node fails collective performance. How should the team decide whether the issue is GPU-to-GPU interconnect versus east-west fabric?

A housing authority multi-GPU node with failed collective performance needs a split diagnosis: compare on-node NVLink/NVSwitch health and bandwidth against fabric link and congestion metrics for cross-node paths. If NVLink errors, degraded bandwidth, or bad topology explain local ranks, fix the GPU interconnect first. If local interconnect looks healthy but multi-node segments show Spectrum/InfiniBand drops, pauses, or hot links, treat it as an east-west fabric problem. That comparison classifies the fault instead of guessing.

Correlating GPU SM utilization and CUDA kernel timing across ranks can localize stall symptoms in a profile, but it does not compare on-node NVLink/NVSwitch health against fabric congestion, so it cannot by itself decide whether the fault is GPU-to-GPU interconnect or east-west fabric. Inspecting Spectrum leaf interface errors and ECN marks looks only at the fabric side during congestion; without parallel on-node NVLink/NVSwitch measurements, it cannot distinguish an interconnect fault from an east-west fabric fault. Benchmarking NCCL bus bandwidth while tuning IB HCA and PCIe affinity changes multiple variables at once and measures aggregate throughput rather than isolating whether collectives stall on the node interconnect or the cross-node fabric.

Exam caveat: isolation order is measure both domains, then act—not correlate GPU SM and kernel timing to localize stalls, inspect Spectrum leaf errors and ECN marks without on-node counters, or benchmark NCCL with IB HCA and PCIe affinity tuning as a substitute for comparing domains.

Operational check: pull NVLink/NVSwitch counters and bandwidth tests on the housing-authority node in parallel with fabric interface, queue, and congestion telemetry for the job’s cross-node flows; ticket the failing domain with evidence, and keep monitoring enabled throughout.