Emergency container logs show OOM kills after Docker memory limits were copied from CPU-only services. What should ops do?
Select an answer to reveal the explanation.
Short Explanation
When you see OOM kills after CPU-only memory caps were copied, give the municipal GPU containers room to breathe by retuning their Docker memory limits. You don’t need to hide the logs or move every emergency pod onto the head node.
Full Explanation
Emergency container logs showing OOM kills after Docker memory limits were copied from CPU-only services call for retuning those limits for municipal GPU containers. GPU workloads often need more host RAM for dataset staging, CUDA context, and framework overhead, even when model weights sit in HBM; CPU-only caps that work for lightweight APIs can kill emergency AI workers under load. Right-sizing memory limits and reservations can prevent OOM while preserving cgroup isolation. Disabling OOM logging fails because it hides the failure signal without adding memory headroom. Lowering GPU counts fails because it reduces capacity and may still leave a process over its memory cap; it treats a limit misconfiguration as a GPU inventory problem. Pinning every emergency pod to the BCM head node fails because head nodes are for control-plane functions, not a dumping ground for memory-hungry workers, and concentration creates a single point of failure. Exam caveat: GPU memory (HBM) exhaustion and host cgroup OOM are different failure modes. Because the stem identifies Docker memory limits copied from CPU-only services, address host memory cgroup limits rather than treating MIG or GPU VRAM as the first issue. Avoid infinite limits as a permanent policy; use evidence-based caps with headroom for peak emergency load. Practical check: correlate OOM killer events with cgroup limit settings for the emergency units, raise limits based on observed RSS and peak use during drills, restart services, and rerun a load test. Confirm OOM counters stay flat and document a GPU-specific limit template so CPU service defaults are not copied blindly.