A housing-eligibility model's SageMaker training job runs on an on-demand training instance and needs to persist checkpoints during the run so training can resume if the job is interrupted. Which storage choice is appropriate for these checkpoints during and after the run?

SageMaker training instances are ephemeral: their attached local storage is torn down with the instance once the job ends or is interrupted, so checkpoints that need to survive an interruption or feed a later resumed run must land somewhere durable — Amazon S3 — even though writing to fast local storage during the run and periodically syncing to S3 is the practical pattern, since writing every checkpoint straight to S3 can add latency the training loop doesn't need. Relying solely on the attached EBS volume assumes that volume persists after the job ends, which it does not for a standard on-demand training job; the instance and its storage are reclaimed, taking any checkpoint that was never synced elsewhere with it. Writing checkpoints to the SageMaker Model Registry misapplies that service — the Model Registry tracks versioned, typically finished model artifacts for governance and deployment, it is not built to receive frequent in-progress checkpoint writes during an active training loop. Skipping checkpointing and relying on a bare restart-from-scratch retry wastes all training progress made before an interruption, which matters for longer or costlier training jobs where redoing hours of work is expensive. Scope note: for spot-based training specifically, frequent checkpointing to S3 becomes even more important given the higher interruption likelihood. Operational check: manually interrupt a test training run and confirm it resumes correctly from the last S3-synced checkpoint before trusting the pattern in production.