A public-health InfiniBand fabric looks link-up but jobs still fail. What should operators interpret?
Select an answer to reveal the explanation.
Short Explanation
Green link lights can still hide a cranky fabric service. Read UFM system health for degraded InfiniBand services beyond simple link-up on the public-health fabric.
Full Explanation
A public-health InfiniBand fabric that looks link-up while jobs still fail requires interpreting UFM system health for degraded fabric services beyond simple link-up status. Physical port green does not guarantee healthy SM, HA, or other fabric services that municipal AI workloads depend on; operators must read system health as a separate plane from L1 link state.
Why the other options fail by concept: assuming port-level link-up status indicating normal NCCL collective bandwidth behavior fails because L1 link-up is a physical-layer signal, while achieved collective bandwidth also depends on SM/HA and other fabric services; a fully link-up fabric can still under-deliver bandwidth when those services are degraded. Assuming vendor InfiniBand specification bandwidth indicating realistic training throughput levels fails because datasheet numbers describe ideal lab conditions, not this public-health fabric's live service health; matching a spec sheet proves nothing about whether UFM-visible services are currently degraded. Treating host ibstat output reporting normal local link rate and link state as sufficient for determining fabric-level service health fails because ibstat reports local port and link state on one host; it does not surface SM/HA or other fabric-wide service degradation that UFM system health is built to show.
Exam caveat: degraded UFM health components may not always drop links; jobs can fail with clean optics. Conversely, a red link is still a data-plane fault—use UFM to scope which domain is unhealthy rather than assuming every job failure is SM-related. Confirm health against the same UFM instance that manages the public-health fabric.
Operational check: when public-health IB jobs fail despite link-up, open UFM system health, identify degraded services or components, correlate with the job-failure window, capture the health export in the ticket, and remediate the unhealthy service domain before mass host reboots.