Long council transcripts exceed one GPU’s activation memory for context. Which approach helps?
Select an answer to reveal the explanation.
Short Explanation
Council minutes are long; one GPU’s activation budget cannot hold the whole context. Sequence parallelism slices the sequence across cards so the long transcript still trains.
Full Explanation
Long council transcripts that exceed one GPU’s activation memory for context are addressed with sequence parallelism: the long sequence (and associated activations/attention state) is split across GPUs so no single device must hold the full context footprint. Shortening every transcript to one sentence before training discards the long-context civic signal the model is supposed to learn and is not a parallelism approach. Relying only on data parallelism without splitting sequence state still leaves each replica responsible for the full activation memory of the long transcript, so the OOM remains. Storing full activations on CPU for every attention layer forever trades GPU OOM for severe PCIe thrash and is a poor substitute for sequence sharding across accelerators. Exam caveat: activation memory for long context is a sequence-dimension problem; solve it by partitioning sequence state, not by deleting the corpus or hoping DP alone helps. Operational check: enable sequence (or context) parallel degree matching transcript length needs, validate attention correctness on known council excerpts, monitor activation memory per rank, and ensure the municipal long-context run completes without host offload as the primary strategy.