Parks computer-vision InfiniBand jobs time out while one spine port collapses in bandwidth. What should operators connect?
Select an answer to reveal the explanation.
Short Explanation
When the vision jobs stall, peek at the fabric speedometer. Tie those timeouts to the UFM bandwidth collapse on that InfiniBand spine port.
Full Explanation
When Parks computer-vision InfiniBand jobs time out while a spine port collapses in bandwidth, the correct operational move is to connect the application symptom to UFM bandwidth utilization on that specific spine port. UFM exposes fabric-level throughput and congestion indicators that map timeouts to a concrete InfiniBand path. Municipal operators can then prove the vision pipeline is stalling because the fabric cannot sustain RDMA rates on the implicated spine, prioritize optics or switch remediation, and avoid mis-scoping the outage to the container image or storage tier.
Correlating NCCL collective timeout warnings with GPU power draw trends in IPMI logs fails because GPU power trends do not explain a concurrent spine bandwidth collapse visible in the fabric; skipping the fabric check leaves the root cause untested. Linking leaf port unicast throughput peaks to container image pull latency on storage fails by asserting a single-domain cause without evidence and by discarding the tool that would falsify that claim. Matching subnet manager routing updates on leaf switches to job scheduler queue wait times fails as a blind distraction: scheduler correlation may briefly obscure symptoms while the spine bottleneck remains.
Exam caveat: Correlation is not automatic causation. A bandwidth drop on a spine may be shared by many jobs; confirm the timed-out Parks workloads traverse that port via topology and job placement before declaring the spine the sole cause.
Operational check: Open UFM bandwidth utilization for the suspect spine during a Parks CV job run, record the drop timeline against job timeout timestamps, identify affected links and HCAs, and only then take port or node recovery actions. Document the correlation so future municipal incidents start with fabric bandwidth review instead of matching leaf-switch routing updates to scheduler queue waits, which would miss the spine-port collapse.