Emergency-management inference pods die intermittently with PCIe error storms under load. What troubleshooting focus is required?
Select an answer to reveal the explanation.
Short Explanation
Intermittent PCIe hiccups under load are like a loose hitch that only shows on the highway—chase them while the system is working hard. Emergency management needs that load-time triage to find the bad hardware path.
Full Explanation
Emergency-management inference pods that die intermittently with PCIe error storms under load require triage of those PCIe errors while the civic workload is stressing the path, so operators can locate the failing hardware link, slot, or device. Domain 5 expects hardware faults that appear only under load to be chased under load, not silenced. Assuming model weights are corrupt and retraining overnight addresses ML artifacts, not PCIe AER, link training, or device faults on the host that kill pods. Disabling PCIe error logging to stabilize pods removes the primary evidence channel and leaves the intermittent FRU undetected until a worse outage. Moving only the hostname into a new DNS zone has no relationship to PCIe fabric integrity on the inference host. Exam caveat: intermittent PCIe may need both idle and peak-load captures because some Correctable/Uncorrectable counters only climb when emergency inference concurrency peaks. Operational check: reproduce a representative emergency inference load on the failing node, capture dmesg/AER and vendor PCIe diagnostics during the storm, map the BDF/slot to a FRU, and validate after reseat or replacement that pods survive the same load profile.