City DGX fine-tuning of a large ordinance model no longer fits DDP replicas per GPU. What multi-GPU choice fits?
Select an answer to reveal the explanation.
Short Explanation
When each GPU cannot hold a full ordinance replica anymore, DDP is out of gas—FSDP shards the weights so the city job still trains. Pretending OOM is fine just stops the run.
Full Explanation
When city DGX fine-tuning of a large ordinance model no longer fits full DDP replicas per GPU, the appropriate multi-GPU choice is Fully Sharded Data Parallel (FSDP), which shards parameters (and typically gradients/optimizer state) across ranks so each GPU holds only a fraction of the model. Classic DDP assumes every rank can store a complete replica; ignoring out-of-memory errors and keeping full replicas simply aborts the municipal job. Switching the entire ordinance stack to CPU-only training abandons GPU throughput and is not a serious multi-GPU strategy for large models. Duplicating the model on every GPU without sharding parameters is still DDP-style replication and does not solve the per-GPU footprint problem that triggered the OOM. Exam caveat: distinguish data-parallel replication (DDP) from sharded data parallel (FSDP/ZeRO-style)—when replica size exceeds device memory, sharding is required, not more copies. Operational check: measure peak allocated memory under DDP, enable FSDP wrapping for transformer blocks when replicas no longer fit, validate loss curves against a smaller baseline, and monitor all-gather/reduce-scatter overhead so ordinance training still meets the city's wall-clock budget.