After an all-reduce stall on the municipal AI fabric, engineers suspect Priority Flow Control contributed to a pause-frame deadlock. What operational step best hardens the civic leaf-spine for future trainer bursts?
Select an answer to reveal the explanation.
Short Explanation
PFC is like traffic lights that can get stuck red if the timing is wrong. After that all-reduce freeze, you keep the lights (lossless priorities) but fix the watchdog so a stuck pause doesn't lock the whole civic fabric.
Full Explanation
Priority Flow Control remains appropriate for lossless RDMA on the municipal AI fabric, but prolonged pause loops can contribute to deadlock-like stalls that freeze all-reduce and similar collectives. The hardening step after a suspected PFC-related stall is to tune deadlock-avoidance and watchdog behaviors so the civic leaf-spine detects and clears stuck pause conditions without abandoning the lossless priorities the trainers still need. That keeps Ethernet RDMA viable while reducing the chance that a pause storm locks the POD. Enabling ECN marking with DCQCN-style rate reduction on RoCE queues signals congestion early but does not detect or clear prolonged PFC pause loops, so it leaves the watchdog and deadlock-avoidance gap unaddressed after a suspected pause-frame stall. Expanding headroom buffers and raising PFC trigger thresholds may absorb microbursts, but higher thresholds delay pause signaling and can deepen congestion before pauses trigger, increasing the risk of stuck pause conditions rather than clearing them. Rebalancing ECMP hashing with flowlet-aware adaptive routing spreads elephant flows across spine links, yet it does not address the PFC pause deadlock itself and offers no watchdog to detect or clear prolonged pause loops on lossless priorities. Exam caveat: watchdog and deadlock-avoidance settings must align with vendor guidance for the Nexus platform and the specific PFC-enabled Cos map; copying generic timer values without testing can still leave trainers exposed. Operational check: reproduce a controlled pause stress test on a lab leaf pair, verify watchdog clears prolonged pause loops, confirm RDMA priorities remain lossless after recovery, and capture pause/deadlock counters in municipal telemetry dashboards.
Prefer to study offline? Printable PDF study book + test engine ($7.50, unofficial). These questions stay free here.