Housing trainers stage a deep encoder across four GPUs while microbatches keep flowing. What technique is that?
Select an answer to reveal the explanation.
Short Explanation
You can picture an assembly line: each GPU owns a stage of the deep encoder, and microbatches move from stage to stage. That’s pipeline parallelism, not single-GPU training with the other accelerators idle.
Full Explanation
Housing trainers who stage a deep encoder across four GPUs while microbatches keep flowing are using pipeline parallelism: successive encoder stages are assigned to successive GPUs, and microbatches move through those stages so devices can stay utilized rather than waiting on a single monolithic forward pass. The key clue is the combination of stages spread across GPUs and continuous microbatch flow. Sequence parallelism, as described here, shards microbatches across GPUs rather than assigning encoder stages, so it does not match the staged execution in the question. Single-GPU training with unused idle accelerators wastes the four-GPU allocation and does not implement the described distributed staging. Expert parallelism is for placing experts in a mixture-of-experts model; it does not describe distributing a dense, non-MoE encoder by stage. Exam caveat: focus on what the question says is assigned to each GPU—encoder stages indicate pipeline parallelism, while distributing microbatches without assigning stages indicates sequence parallelism. Operational check: partition the encoder into balanced stages, choose enough microbatches to keep the stages occupied while limiting pipeline bubbles, and monitor stage idle time and activation memory. On a small run, confirm that end-to-end loss matches a single-GPU reference before scaling up housing training.