A 311 model-farm runbook must escalate InfiniBand connectivity checks in a sensible order. Which sequence is appropriate?

A 311 model-farm InfiniBand runbook should escalate connectivity checks from local state to path reachability, fabric diagnosis, then performance quantification. ibstat first establishes HCA and port state on the municipal GPU host. ibping next verifies InfiniBand reachability toward a peer. ibdiagnet then sweeps fabric-level anomalies when the problem is broader than one hop. ib_write_lat or ib_write_bw finally measures latency or bandwidth once basic connectivity is believable, so 311 operators quantify residual performance issues instead of guessing. Starting with ib_write_lat or ib_write_bw, then running ibdiagnet, then ibping, and leaving ibstat for last inverts the order: performance tests and fabric sweeps before local/link state are hard to interpret, and checking the HCA only at the end means you miss the most basic failure point first. Running ibdiagnet first, then ib_write_lat or ib_write_bw, then ibping, and leaving ibstat for last skips local-state and reachability checks until the end, so you sweep the fabric and measure performance before knowing whether the HCA or port is even healthy. Using ibstat, then ibnetdiscover, then ibping, then ib_write_lat or ib_write_bw skips the ibdiagnet fabric sweep entirely, so you move from local state and discovery to performance tests without ever checking for fabric-level anomalies. Exam caveat: escalate only as far as symptoms require—if ibstat shows the port Down, fix that before a full fabric sweep or bandwidth campaign. Operational check: follow ibstat → ibping → ibdiagnet → ib_write_lat/ib_write_bw on the 311 model farm, record outputs at each stage, and stop escalating once the failing layer is identified and remediated.