Library NLP runs multi-hour NCCL burn-in and needs early warning of latent GPU faults. What should operators capture?

Library NLP multi-hour NCCL burn-in should log thermal and error trends for each GPU node throughout the run so operators get early warning of latent faults before a final pass/fail bit hides the story. Progressive temperature climb, rising XID counts, or link flaps often whisper for hours before a hard failure, and Domain 4 observability is how municipal teams catch that progression. Capturing only the final pass/fail bit with no intermediate telemetry discards the diagnostic timeline needed to decide whether a node is marginal or healthy. Screenshotting the cafeteria menu as the burn-in artifact is irrelevant theater and supplies no GPU or fabric evidence. Disabling logging to reduce I/O during burn-in removes the very signals burn-in is meant to produce and undermines later root-cause analysis. Exam caveat: logging should be sized so it does not itself saturate the training filesystem—use node-local or dedicated logging paths with rotation, but do not confuse “manage log volume” with “turn logging off.” Operational check: for library NLP burn-in, enable per-GPU thermal and error trend capture for the full window, review mid-run and end-of-run charts against policy, and quarantine any node whose trends breach thresholds even if the last NCCL iteration still completed.