A transit ML cluster enables RoCE on Spectrum-X switches but training still stalls under load. Host PFC and priority settings were never aligned. What is the most likely gap?
Select an answer to reveal the explanation.
Short Explanation
Turning on RoCE only on the Spectrum-X switches is like painting bike lanes on the avenue but forgetting the curb cuts at every garage—the transit GPUs still need matching PFC and priority on the hosts. Lossless RoCE is a team sport between switch and server.
Full Explanation
When a transit ML cluster enables RoCE on Spectrum-X switches but training still stalls under load, and host PFC and priority settings were never aligned, the most likely gap is missing end-to-end coordination: Spectrum-X RoCE requires lossless settings on both switches and hosts, not switch-only enablement. Priority flow control, DSCP/priority mapping, and related QoS must match from NIC through leaf so RoCE queues remain lossless under congestion. Switch-side RoCE without host PFC/priority alignment leaves pause and marking behavior incomplete, so collectives stall when the path is stressed.
Assuming switch-side buffer allocation alone handles RoCE while host priority mapping stays at defaults contradicts lossless RoCE practice on Spectrum-X. Blaming stalls on spine uplink congestion and calling for more spine capacity misses the misaligned host PFC and priority settings that actually cause the stall. Jumbo MTU alignment on Spectrum-X switches by itself does not deliver lossless RoCE transport during congestion without coordinated host settings.
Exam caveat: RoCE is an end-to-end contract; exam items often punish "enabled on the switch only." Operational check: on the transit hosts, verify PFC enabled on the RoCE priorities, match Spectrum-X QoS/priority maps and MTU, generate load, and watch for stalls clearing; do not declare victory from switch CLI alone, leave host priority at defaults while tuning switch buffers, add spine capacity as the first response, or treat jumbo MTU as a standalone lossless fix.