County health NCCL burn-in accumulates XID and link errors past policy thresholds. What should operators do?
Select an answer to reveal the explanation.
Short Explanation
When XID and link errors exceed policy thresholds, you should stop the county health burn-in and fail the affected nodes. That protects the integrity of your results instead of hiding faults behind a polished report.
Full Explanation
When county health NCCL burn-in accumulates XID and link errors beyond policy thresholds, you should abort the run and fail the offending nodes. Burn-in is intended to expose latent GPU and fabric faults, so crossing the error budget is a failure condition—not a report-formatting issue. Muting alerts and finishing the run for a polished report conceals unstable GPUs or links and can let unreliable capacity enter the municipal pool. Raising thresholds mid-run changes the acceptance criteria after testing has begun, undermining the policy and making the results meaningless. Ignoring XIDs on the grounds that they matter only for desktop GPUs is also incorrect: XID and link errors during collective burn-in are important data-center signals. Exam caveat: distinguish a transient, logged single event from accumulated errors that cross the written abort threshold; follow the established policy rather than inventing limits during a test. Practical check: define XID and link-error abort criteria before burn-in, monitor counts as the run proceeds, stop when a threshold is crossed, quarantine failed nodes from training queues, and remediate before retesting.