A school-district IT GPU lab asks when GPU-to-GPU traffic should stay on NVLink/NVSwitch versus traverse the Spectrum or InfiniBand fabric. Which guidance is correct?
Select an answer to reveal the explanation.
Short Explanation
Inside one multi-GPU box, NVLink/NVSwitch is the express elevator between floors; the Spectrum or InfiniBand fabric is the street network between buildings. School-district labs should keep on-node GPU talk on NVLink when they have it, and reserve the fabric for jobs that actually span nodes.
Full Explanation
In a school-district IT GPU lab, intra-node GPU-to-GPU traffic should stay on NVLink/NVSwitch when those links are available, while multi-node distributed communication should use the Spectrum or InfiniBand fabric. NVLink/NVSwitch provides the highest bandwidth and lowest latency path inside a multi-GPU server; forcing that traffic onto the external fabric wastes fabric capacity and adds hops. Preferring the Spectrum or InfiniBand fabric for intra-node collectives while reserving NVLink for cross-node synchronization is wrong: it throws away the fastest local path for on-node traffic and misuses NVLink, which cannot span nodes. Extending NVLink/NVSwitch between nodes for distributed jobs and bypassing the Spectrum or InfiniBand fabric is wrong: NVLink is not designed to span nodes, and multi-node collectives need the data-center fabric. Routing intra-node GPU-to-GPU collectives over Spectrum Ethernet and keeping NVLink only as a backup failover path is wrong: it relegates the highest-bandwidth local link to standby and forces GPU traffic through slower Ethernet for monitoring convenience.
Exam caveat: the decision boundary is domain scope—same node versus multi-node—not "always fabric for intra-node traffic" or "always NVLink across nodes." Operational check: on a district multi-GPU node, confirm NVLink/NVSwitch topology is healthy (nvidia-smi topo / NVLink counters), ensure NCCL prefers NVLink for intra-node pairs rather than Spectrum Ethernet, and confirm cross-node jobs land on Spectrum or InfiniBand data ports rather than attempting to extend NVLink between nodes; document that split in the lab runbook.