Water multi-GPU job sees peer-to-peer disabled; fabric manager health is the fix path. What should ops do?
Select an answer to reveal the explanation.
Short Explanation
Disabled P2P on a water multi-GPU job often means fabric manager health is sick—heal the fabric to reopen NVLink lanes. Doubling walltime forever is surrender, not ops.
Full Explanation
Water multi-GPU jobs that see peer-to-peer disabled should drive fabric manager health troubleshooting to restore P2P and NVLink capability for the civic job. On NVLink/NVSwitch platforms, unhealthy fabric manager commonly coincides with disabled or unavailable P2P paths, forcing slow fallback copies and crushing multi-GPU scaling for utility models. Restoring fabric health re-enables the peer path the job was sized for.
Accepting disabled P2P and doubling walltime forever fails by concept: it institutionalizes a fabric fault as a scheduling constant and wastes municipal GPU hours. Disabling CUDA entirely on water-utility nodes fails the workload class—no CUDA means no GPU job. Moving multi-GPU training to a single CPU core fails both performance and the multi-GPU requirement; it is not a P2P remediation.
Exam caveat: P2P can also be disabled by topology, ACS, or container device isolation—but when the stem states fabric manager health is the fix path, follow that. Do not confuse "P2P disabled" messages with simple single-GPU jobs that never needed P2P.
Operational check: query P2P/NVLink status and fabric manager health on the water node, remediate fabric manager (service, versions, links), confirm P2P enablement returns, run a multi-GPU bandwidth/NCCL test, and then rerun the utility job to verify step times recover without doubling walltime.