A city GPU fabric replaces a failed InfiniBand switch and must keep subnet manager HA intact. What should operators do?
Select an answer to reveal the explanation.
Short Explanation
Don't elect a new master on the replacement switch and demote healthy SMs—that drops HA redundancy. Don't reset to vendor defaults for testing or re-discover topology after GPU traffic resumes—that skips HA validation. Re-provision the replacement and keep the city GPU fabric's dual subnet manager HA configuration intact.
Full Explanation
When a city GPU fabric replaces a failed InfiniBand switch and must keep subnet manager HA intact, operators should re-provision the InfiniBand switch while preserving high-availability subnet manager configuration. Hardware replacement must restore ports and links without collapsing the dual-SM HA design that protects municipal GPU workloads from SM single points of failure.
Electing a new SM master on the replacement switch and demoting existing healthy SM nodes to standby fails by concept: it removes healthy SMs from active duty and creates a single point of failure, violating HA intent. Restoring switch firmware from vendor defaults and configuring SM with lower priority for testing fails because it resets switch-level HA state and can renumber or destabilize the existing civic IB domain. Re-discovering InfiniBand fabric topology from the replacement switch and merging SM state after GPU traffic resumes fails: hosts should not join until the replacement switch and SM HA are verified, and merging state after traffic resumes risks fabric inconsistency.
Exam caveat: IB switch RMA/replacement preserves HA subnet manager configuration; do not elect a new SM master on the replacement switch and demote healthy SMs, restore vendor defaults with a test SM priority, or re-discover topology and merge SM state after GPU traffic resumes.
Operational check: replace the failed city IB switch, re-apply port/GUID provisioning as required, confirm both HA SMs remain configured and healthy, run a brief SM failover or status check, then re-attach GPU links and validate municipal AI jobs on the restored fabric.