Transit checkpoint writes bypass GPUDirect and overload CPUs. What should ops check?
Select an answer to reveal the explanation.
Short Explanation
When transit checkpoint writes overload CPUs, you should confirm Magnum IO acceleration is enabled for that civic checkpoint path. Increasing CPU oversubscription, defaulting all transit output to the BCM head node’s local root disk, or turning off checkpointing won’t restore the accelerated path.
Full Explanation
Transit checkpoint writes that bypass GPUDirect and overload CPUs indicate that Magnum IO acceleration is missing or not engaged for that IO path. Confirming acceleration is enabled for civic checkpoint IO restores GPUDirect-oriented paths and reduces CPU copy overhead during frequent checkpointing. Increasing CPU oversubscription so overloaded hosts look busier worsens contention; it does not restore accelerated checkpoint IO. Configuring all transit checkpoint output to use the BCM head node’s local root disk concentrates storage there, risks filling that disk, and still may leave writes CPU-bound without Magnum IO acceleration. Turning off checkpointing removes the IO load, but also removes fault tolerance that emergency and transit training jobs rely on. The key check is configuration and enablement, not simply reducing checkpoint frequency. Exam caveat: CPU saturation during checkpoint storms can mean the accelerated Magnum IO path was never enabled, rather than checkpoint frequency alone being the problem. Practical check: inspect Magnum IO settings for checkpoint directories on transit training nodes, enable acceleration per civic standard, then write a test checkpoint while monitoring CPU use and GPUDirect metrics. Confirm CPU copy overhead drops before returning production training to its normal checkpoint cadence.