During a months-long training run of a foundation model, the system periodically saves large model checkpoint files. These files are massive, written sequentially, read only during crash recovery, and do not need a hierarchical directory structure. Which storage type is the most cost-effective and appropriate for archiving these checkpoint files?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of checkpointing like saving your game progress in a massive RPG. Every few hours, you want to save the entire state of your model—which could be hundreds of gigabytes—so if a GPU burns out or the power cuts, you don't lose weeks of work. These checkpoint files are huge, write-once, and read-almost-never (unless things break). Now, your boss might say, "Let's put everything on our blazing-fast parallel file system!" But those systems are expensive. Using Lustre or GPFS for long-term archiving of cold checkpoint files is like buying a Ferrari just to store cardboard boxes in the back seat. Object storage—like AWS S3 or Ceph—is perfect here. It's incredibly cheap, handles massive files easily, and since we don't need complex folder structures or fast random reads, it saves you a ton of cash. Trust me on this one, object storage is the champion for checkpoint archiving.
Full explanation below image
Full Explanation
Model checkpoints are copies of the complete state of a model (including weights, biases, optimizer states, and metadata) saved at specific intervals during training. This ensures that if a hardware failure occurs in a multi-node cluster, training can resume from the last saved state rather than restarting from scratch. These checkpoint files have distinct storage characteristics: large file sizes, write-once/read-rarely patterns, and flat structure (no POSIX directory requirements). Object storage (such as AWS S3, Google Cloud Storage, Azure Blob Storage, or on-premises Ceph) is the most cost-effective solution for this workload. It offers highly durable, virtually unlimited scale-out capacity at a fraction of the cost of block or file storage. It uses a flat namespace and simple API-driven access (PUT/GET), which perfectly aligns with asynchronous checkpointing. Let's look at why the others are less suitable: Option A (Parallel File Systems like Lustre or GPFS) is extremely fast but highly expensive. It is best reserved for the active, hot training loop where active data loading requires high-throughput, low-latency random reads. Option B (Block Storage / NVMe-oF) is designed for low-latency database or OS workloads and is cost-prohibitive for archiving terabytes of sequential files. Option C (NFS) suffers from scaling limitations and performance bottlenecks when multiple nodes attempt to write massive files simultaneously over a shared mount point.