A public-health GPU hall checkpoints training every few minutes and needs high IOPS NVMe so writes do not stall the GPUs. Which storage evaluation focus is correct?

A public-health GPU hall that checkpoints training every few minutes places a sustained write burden on storage. Correct evaluation prioritizes storage performance—IOPS and latency—sized so those frequent checkpoint writes complete without stalling GPUs. NVMe-class media on the hot path is the typical match because mid-epoch saves are latency- and IOPS-sensitive; when checkpoints lag, trainers block, iteration time stretches, and expensive accelerators sit underutilized.

Prioritizing sequential throughput (GB/s bandwidth) sized for frequent checkpoint writes fails by concept: high streaming bandwidth does not guarantee the write IOPS and low p99 latency that concurrent mid-epoch saves need, so trainers can still block while waiting for many small checkpoint commits to complete. Prioritizing read-cache IOPS (NVMe latency) sized for frequent checkpoint writes fails by concept: read-cache hits accelerate reads, not the sustained write burden of checkpoint persistence, so GPUs still stall when frequent mid-epoch writes wait on backend commit latency. Prioritizing backend NVMe IOPS (write latency) sized for infrequent checkpoint restores misses the use case: sizing for infrequent restores provisions for the rare read-back path and leaves no headroom for the write IOPS and latency a live checkpoint needs every few minutes.

Exam caveat: Domain 2.3 requires matching storage performance characteristics to AI I/O patterns. Capacity and archival tiers remain necessary for finished models and datasets, but they do not substitute for the hot checkpoint path described here.

Operational check: measure checkpoint write size and frequency under representative public-health jobs; confirm NVMe (or equivalent) delivers headroom on IOPS and p99 latency during concurrent saves; alert when GPU wait-on-I/O rises so storage, not model code, is investigated first.