School-district lab sees OOM and faults only during NeMo burn-in after HPL/NCCL passed. What should operators do?
Select an answer to reveal the explanation.
Short Explanation
If NeMo burn-in alone makes a school-lab node OOM or fault, that node flunks—green HPL does not get it a free pass into production.
Full Explanation
When a school-district lab node OOMs or faults only during NeMo burn-in after HPL and NCCL passed, operators should fail that node and keep it out of production until remediated. Domain 4 treats framework-unique failures as actionable because production AI jobs exercise those same paths; green microbenchmarks do not grant a free pass. Accepting the node because microbenchmarks were green imports a latent training-stack defect into the student or research GPU pool. Disabling NeMo burn-in permanently to avoid awkward failures removes the detection method instead of fixing memory, driver, or image issues the framework run exposed. Blaming the users and ignoring the burn-in evidence rejects objective test output and violates municipal verification discipline. Exam caveat: confirm the NeMo config was appropriately sized for the GPU—true defects still fail a fair config, but an intentionally oversized model can OOM healthy nodes and should be corrected before condemning hardware. Operational check: quarantine the school-lab node that failed NeMo-only, capture OOM/fault logs with GPU memory telemetry, remediate (memory RMA, driver/image fix, or bad GPU), re-run NeMo burn-in, and only then return the node to the production-capable pool.