Transit ops replaces a failed NIC on a GPU training node. What must happen before returning the node to the pool?
Select an answer to reveal the explanation.
Short Explanation
A detected NIC is a good start, but you still need to prove the data path works before the node rejoins the pool. Re-check link health and NCCL behavior so you can trust it in training again.
Full Explanation
After transit ops replaces a failed NIC on a GPU training node, the node must not return to the pool until link health and NCCL behavior are re-validated. This post-replacement proof confirms that the new FRU restored fabric participation for municipal training collectives, not merely that the card is seated electrically or appears in the operating system. Latent bad optics, incomplete seating, or a firmware mismatch can go unnoticed yet still break all-reduce under load. Returning the node once the operating system detects the NIC checks enumeration, not data-plane health, and risks reintroducing collective failures into the transit training queue. Skipping NCCL because HPL passed last quarter relies on stale, possibly different stress coverage and does not test the replaced adapter in multi-GPU communication. Pinging the BMC verifies out-of-band manageability, not GPU data-plane links or NCCL, so it cannot establish fabric readiness. Exam caveat: validation should include the same rail or fabric peers the node uses in production, not just a lone loopback or management ping. Practical check: verify data-fabric link status, speed, and optics, run bandwidth and NCCL smoke tests with peer GPU nodes, and only then re-enable the transit node in the scheduler pool.