Transit ETA training checkpoints crawl across the network. What architecture change helps?

Transit ETA training writes large checkpoints frequently. When object storage is far from the GPU rows, each save incurs high RTT and competes for WAN and core-link capacity. Colocating high-throughput storage with the GPU rows shortens the path, increases checkpoint bandwidth, and helps municipal ETA training avoid leaving GPUs idle while waiting on distant storage; the architectural principle is locating persistence near compute. Moving all checkpoints to removable USB sticks at depots is not scalable or suitable for concurrent cluster access, and does not provide a cluster filesystem. Placing object storage only in a distant cold site with no local cache maximizes latency for frequent training saves. Deleting checkpoints every minute may reduce traffic, but destroys recoverability without fixing the RTT problem and risks losing training progress. Exam caveat: colocated storage still needs backup and disaster-recovery copies off-row; fast local storage should not be the sole copy, so pair local performance with durable remote retention for civic continuity. Practical check: measure checkpoint write throughput from GPU nodes to the proposed storage, compare it with the distant cold path, and confirm ETA training iteration time improves without deleting recovery points. Local checkpointing can also reduce contention on shared civic WAN links carrying CAD, AVL, and public Open Data feeds, protecting rider-facing services while GPUs stay busy. Use asynchronous replication to the cold site rather than sending every fsync across the metro.