A dense civic training node shows GPUs isolated after maintenance because fabric manager is stopped. What should ops do?
Select an answer to reveal the explanation.
Short Explanation
Isolated GPUs after maintenance often mean fabric manager is down—bring the fabric service back like reconnecting the subway lines. Assuming isolation is forever skips the easy win.
Full Explanation
A dense civic training node that shows GPUs isolated after maintenance because fabric manager is stopped needs fabric manager restarted and repaired so NVLink/NVSwitch topology returns for municipal training. Fabric manager programs and supervises the NVSwitch fabric; when it is down, GPUs may appear as islands without the peer connectivity NCCL expects. Restoring the service re-enables the topology dense training depends on.
Assuming isolated GPUs are a desirable permanent state fails by concept: isolation after maintenance is a fault symptom on NVSwitch systems, not a performance feature. Permanently disabling NVLink in every application fails the dense-node design and locks in slow paths. Replacing the entire rack before checking fabric manager status fails troubleshooting order—service state is cheaper and far more likely after maintenance than simultaneous hardware death across the rack.
Exam caveat: fabric manager is relevant on NVSwitch/NVLink fabric systems; do not assume it is required on every single-GPU civic node. Also distinguish "GPUs visible in nvidia-smi but P2P/NVLink missing" from "driver not loaded." The stem says fabric manager is stopped—start and stabilize that service first.
Operational check: check fabric manager status/logs, start/enable the unit, verify topology tools show restored NVSwitch links, run an NCCL all-reduce smoke on the dense node, and add a post-maintenance checklist item so fabric manager health is verified before declaring the civic training node ready.