A long-context ordinance model must spread QKV GEMMs across GPUs. What distribution approach fits?
Select an answer to reveal the explanation.
Short Explanation
For long ordinances, you can spread the QKV workload by sharding attention heads across GPUs, so the GEMMs share the load. Keeping every head on one GPU or running attention on the CPU works against the long-context goal; head sharding is the fit here.
Full Explanation
Multi-GPU attention-head sharding is the right fit because it distributes attention heads and their QKV projections across devices, spreading GEMM work and activation memory instead of concentrating them on one overloaded GPU. This directly meets the requirement to distribute QKV computation for long ordinances. Keeping all attention heads on one GPU leaves the QKV hotspot and memory pressure unmitigated, so it will not scale with transcript or bill length. Sharding only the optimizer state can reduce optimizer memory, as in ZeRO-style approaches, but it does not distribute the attention GEMMs or address the requested QKV computation. Running attention exclusively on the host CPU for long ordinances sacrifices throughput and conflicts with the goal of using multiple GPUs. Exam caveat: long-context bottlenecks often arise in attention, so head sharding addresses a different problem from optimizer-state sharding alone. Practical check: enable attention-head sharding across GPUs, validate long-context correctness on sample bills and amendments, profile QKV time per rank to check load balance, and confirm that interconnect bandwidth lets attention collectives run without dominating step time.