Emergency-management fine-tunes fragment GPU memory mid-run despite average free VRAM looking fine. What should ops do?

Emergency-management fine-tunes that fragment GPU memory mid-run can fail large allocations even when average free VRAM looks comfortable, because free bytes sit in holes too small for the next contiguous tensor. Ops should capture CUDA memory snapshots, inspect the caching allocator’s free-block distribution, and adjust caching settings or reuse patterns—such as pooling, avoiding unnecessary expand/shrink cycles, or enabling expandable segments where supported—so peak requests succeed through the run. Ignoring snapshots and allocating ever-larger tensors until crash merely advances the failure point without diagnosing fragmentation. Switching training to a single CPU core avoids GPU fragmentation only by abandoning accelerator throughput that emergency models need. Storing all activations permanently on disk for every microbatch trades VRAM pressure for severe I/O latency and still does not repair allocator fragmentation for the tensors that remain on device. Exam caveat: snapshot tooling and allocator knobs differ across CUDA and framework versions; a fix validated on one DGX image may need re-checking after a driver or PyTorch upgrade in the emergency ops cluster. Operational check: take a memory snapshot at the first near-OOM, apply allocator or reuse changes, then verify subsequent snapshots show fewer unusable free fragments and that the fine-tune completes a full epoch without mid-run allocation failures.