Public-health checkpoints on spinning disks stall training for hours. What storage remediation fits?
Select an answer to reveal the explanation.
Short Explanation
Spinning disks can turn checkpoints into coffee breaks, so move your public-health training checkpoints to NVMe scratch instead. You’ll keep recovery points without letting slow storage stall training.
Full Explanation
Public-health checkpoints on spinning disks can cause multi-hour stalls because their sequential and random I/O exceeds what HDD tiers can sustain at the training cadence. Moving civic training checkpoints to NVMe scratch addresses this storage-tier mismatch while keeping checkpointing enabled for fault tolerance; NVMe scratch provides the fast write and read performance needed for frequent checkpoints during active municipal training. Keeping checkpoints on the slowest disks to reduce NVMe wear prioritizes a concern about media wear over training service-level objectives and leaves public-health job stalls unresolved. Disabling checkpoints removes recovery points that long municipal training jobs need if they fail. Replicating every checkpoint to tape before each epoch puts archival latency in the training critical path and does not provide a fast checkpoint tier. Exam caveat: tiering is a performance design choice—use fast scratch for active checkpoints and colder storage for longer retention—not disabling checkpoints or sending every epoch to tape. Practical check: relocate a public-health training job’s checkpoint directory to NVMe scratch, time a checkpoint write and read against the prior HDD baseline, confirm epoch time drops, and document copying retained checkpoints to colder municipal storage outside the training critical path.