A county continued-pretrain of a 70B civic model needs layers split across GPUs. Which parallelism applies?
Select an answer to reveal the explanation.
Short Explanation
When your 70B model cannot fit on one GPU, tensor parallelism splits attention and MLP weights across GPUs so training can proceed. Replicating an unfittable model with data parallelism will not solve the memory limit.
Full Explanation
A county continued-pretrain of a 70B civic model that needs layers split across GPUs calls for tensor parallelism: attention and MLP weight matrices are partitioned so each GPU computes a shard of the linear operations and communicates activations or partial results as required by the schedule. This addresses the constraint that layer tensors do not fit on one device. Data parallelism alone replicates the full model on each rank, so if one GPU cannot hold the 70B weights, replication cannot make training start. Expert parallelism is for architectures with MoE experts placed across devices; without MoE experts, it does not partition dense attention or MLP weights. Host RAM paging may limp through a demo, but PCIe traffic can sharply reduce tokens per second, so paging is not a substitute for tensor splits across accelerators on a county training cluster. Exam caveat: identify the memory constraint; an unfittable model replica is not made workable simply by adding data-parallel copies. Practical check: choose a tensor-parallel degree that divides the head count and hidden size cleanly, place ranks on a fast interconnect such as NVLink or NVSwitch, verify numerical parity on a short civic-corpus run against a known baseline, and track all-reduce and all-gather time to keep the continued-pretrain efficient under county SLAs.