Transit ops adds a new GPU rack and needs a reference for later fabric regressions. What NCCL practice is required?
Select an answer to reveal the explanation.
Short Explanation
After a new transit GPU rack lands, write down the NCCL all-reduce bandwidth—future you will need that yardstick when something goes weird.
Full Explanation
After transit ops adds a new GPU rack, Domain 4 requires baselining multi-node NCCL all-reduce bandwidth so later fabric regressions have a comparable reference. A post-add baseline captures how the expanded municipal AI factory actually performs under collectives and becomes the yardstick for troubleshooting when performance drifts. Skipping baselines until the first production outage delays detection until riders or training jobs already feel pain and removes the “known good” comparison that shortens mean time to repair. Baselining only keyboard LEDs on the KVM records console cosmetics, not east-west collective bandwidth, and cannot support fabric capacity analysis. Storing bandwidth numbers only in verbal shift handoffs is ephemeral and non-auditable; transit change control needs written metrics tied to rack ID, date, and topology version. Exam caveat: re-baseline after material fabric changes (new racks, switch replacements, cable plants), not only at initial build—stale baselines from a smaller footprint can falsely “pass” a degraded expanded fabric if you never update them. Operational check: after the new transit GPU rack joins the fabric, run multi-node NCCL all-reduce across old and new nodes, archive bandwidth and scaling results with the change record, and define alert thresholds relative to that baseline for ongoing Domain 4 regression detection.