After BCM head-node failover, Slurm job submits through BCM fail for the transit authority. What should ops repair?
Select an answer to reveal the explanation.
Short Explanation
Failover moved the BCM crown, but HA state got wobbly—fix that so Slurm submits through BCM work for transit again. Nuking node images or parking jobs on a laptop is not HA repair.
Full Explanation
After BCM head-node failover, municipal Slurm submissions that route through BCM can fail when high-availability state is incomplete, stale, or not fully promoted on the surviving head. Repairing BCM HA state restores the control-plane path so transit-authority job submits via BCM succeed again without rebuilding compute nodes. The failure mode is typically authentication, scheduler proxy, or database/session state on the HA pair—not corruption of every GPU image.
Wiping every compute node image because failover always corrupts GPUs over-remediates: failover affects head-node HA services, not wholesale GPU filesystem integrity. Disabling Slurm permanently and running all transit jobs on a single laptop abandons the municipal scheduler and cannot absorb authority-scale batch demand. Turning off BCM HA so future failovers never occur removes resilience; it avoids the symptom by accepting a single point of failure instead of repairing HA state.
Exam caveat: distinguish compute-plane symptoms from control-plane HA breakage. If jobs fail only through BCM after failover while direct node health looks fine, prioritize BCM HA repair over node reimaging. Operational check: verify which BCM head is active after failover, confirm HA database and service state are healthy, submit a low-priority transit Slurm job through BCM, and only escalate to node image work if HA repair leaves submits failing with evidence of compute-side breakage.