Water-lab LLM OOMs only on rare very long reports; memory profiles show activation peaks. What policy fits?
Select an answer to reveal the explanation.
Short Explanation
Those rare, novel-length lab reports can make activation peaks surge like a fire hose into a water balloon. Use truncation or chunking guided by the profile’s peak, so you don’t have to pray a restart happens to fit.
Full Explanation
When your water-lab LLM OOMs only on rare, very long reports and profiles show activation peaks, the problem points to sequence-length-driven activation spikes, not a steady-state weight footprint. Adopt input-length truncation or chunking guided by those peaks: ordinary reports can retain full context, while outliers are capped or segmented before peak activations exceed VRAM. Ignoring rare long reports and repeatedly restarting jobs is not a control policy; a novel-length report can still trigger OOM in production, and a lucky run does not make the workload reliable. Doubling model width increases memory demands from both activations and weights, worsening rather than addressing this failure mode. Turning off memory profiling only hides the peaks from auditors; it does not change the OOM physics. Exam caveat: naive truncation can drop critical findings near the end of a lab narrative, so use overlap-aware chunking when completeness matters. Practical check: use the activation-peak profile to set a maximum token threshold just below the OOM cliff, replay the longest historical reports through the policy, and verify successful allocation and coverage of required sections in reconstructed summaries before production rollout.