An engineering team is preparing to train a massive, multi-billion parameter model whose weights and activation states exceed the physical VRAM capacity of a single GPU. Which distributed training techniques should they employ to overcome these memory limitations? (Select all that apply)
Select all correct answers, then click Submit.
Short Explanation and Infographic
Here's the deal: when you're training a massive model, your biggest enemy is running out of GPU memory (VRAM). If your model is 100 billion parameters, it won't even fit on a single 80GB GPU to start training, let alone leave room for gradients and optimizer states! If you try to use standard Distributed Data Parallel (DDP), you're just replicating the entire model onto every single GPU. If it doesn't fit on one, it won't fit on any of them! That's a direct road to Out of Memory (OOM) errors. Instead, we need to shard or split the model. Check this out: Fully Sharded Data Parallel (FSDP) and DeepSpeed ZeRO are absolute lifesavers here. They break up the model parameters, gradients, and optimizer states and distribute them across all your GPUs, only gathering them when they are needed for a specific step. You can also use Pipeline parallelism to split the model's layers across a chain of GPUs (GPU 1 does layers 1-10, GPU 2 does 11-20, and so on) or Tensor parallelism to split a single matrix multiplication across multiple GPUs. These are the techniques that let us train the massive LLMs we use today!
Full explanation below image
Full Explanation
Distributed training strategies for handling model memory constraints (when the model is too big to fit on a single GPU's VRAM) must bypass simple replication: - Fully Sharded Data Parallel (FSDP) (Option C): FSDP is a technique that shards the model parameters, gradients, and optimizer states across all data-parallel ranks. This dramatically reduces the memory footprint per GPU, allowing models that would otherwise cause OOM errors to be trained by gathering weights dynamically during the forward and backward passes. - DeepSpeed ZeRO (Option D): The Zero Redundancy Optimizer (ZeRO) eliminates memory redundancies in data-parallel training by partitioning optimizer states, gradients, and model parameters across GPUs (ZeRO Stage 1, 2, and 3, respectively) rather than replicating them. - Pipeline Parallelism (Option E): This strategy splits the model layers chronologically/sequentially across different GPUs or nodes. For example, if a model has 80 layers, node 1 handles layers 1–20, node 2 handles 21–40, and so on. This divides the memory requirements of both parameters and activation states. - Model/Tensor Parallelism (Option F): This strategy parallelizes the intra-layer computations (such as the large matrix multiplications inside a Transformer's multi-head attention block) across multiple GPUs, dividing the memory required for weights and activations within a single layer. Let's analyze the incorrect options: - Option A is incorrect because standard Distributed Data Parallel (DDP) requires every participating GPU to load a complete replica of the model, optimizer states, and gradients. Therefore, if the model exceeds the memory of a single GPU, DDP cannot be used because it will immediately trigger OOM errors. - Option B is incorrect because increasing the batch size in a basic data-parallel configuration increases memory usage because more activation data must be stored during the forward pass. This worsens, rather than solves, the memory constraint.