Transit ML operators want warning before InfiniBand jobs fail on a dead link. What should they configure?
Select an answer to reveal the explanation.
Short Explanation
Better a UFM ping than a failed all-reduce at 2 a.m. Configure UFM monitoring and alerting so transit operators catch InfiniBand link-down events before jobs die.
Full Explanation
Transit ML operators who want warning before InfiniBand jobs fail on a dead link should configure UFM monitoring and alerting for InfiniBand link-down events. Proactive alerts on link-down and related health events give on-call time to drain jobs, fail over paths, or dispatch repair before collectives time out and overnight training fails. That early-warning pattern is how municipal transit NOCs keep IB link health visible ahead of job loss.
Configuring no alerts and waiting until training jobs time out fails by concept because detection after failure is the opposite of early warning and burns GPU and analyst time. Alerting only after deliberately powering off every switch fails because that is a destructive drill, not an operational notification for production single-link faults. Email alerts for Ethernet DHCP renewals instead of InfiniBand link health fail because DHCP lease noise does not indicate IB link-down on the training fabric and trains operators on the wrong signal.
Exam caveat: useful alerts are fabric-specific—Ethernet management events are not a proxy for InfiniBand link-down, and alert storms without clear IB link context also fail operationally. Operational check: enable UFM link-down alerting to the transit on-call channel, carefully test a single link event in a maintenance window, confirm notification arrives with enough lead time to act, and verify the alert names the IB link rather than an unrelated Ethernet event.