Troubleshooting and Optimization

NCP-AIO · 46 questions

• 311 inference pods CrashLoop after a toolkit misconfig and Docker cannot see GPUs. What should ops troubleshoot first?
• Public-health build cache fills disk and civic nodes run out of space. What Docker remediation fits?
• Transit containers cannot pull because air-gap registry mirrors on the Docker daemon are wrong. What should ops fix?
• Library CV container runs on CPU only because GPU request flags were omitted in the unit file. What fix fits?
• Courts OCR workers die with permission errors on NVIDIA device nodes. What Docker troubleshooting step fits?
• Water-utility hosts run an old Docker engine that rejects the nvidia runtime required by the new stack. What should ops recognize?
• School-lab containers inherit the wrong default runtime after a partial toolkit install. What should ops repair?
• Emergency container logs show OOM kills after Docker memory limits were copied from CPU-only services. What should ops do?
• City nodes have duplicate Docker and containerd paths and Kubernetes calls the wrong runtime. What should ops untangle?
• A dense civic training node shows GPUs isolated after maintenance because fabric manager is stopped. What should ops do?
• Public-health all-reduce hangs and topology tools show a broken NVSwitch path while fabric manager logs confirm issues. What triage fits?
• Transit dense GPU node fails fabric manager on boot after a BIOS change. What should ops do?
• County training slowdown correlates with fabric manager in a degraded state, not application code. What should ops conclude?
• Library mistakenly installs fabric manager on non-NVSwitch nodes and creates service noise. What should ops know?
• Water multi-GPU job sees peer-to-peer disabled; fabric manager health is the fix path. What should ops do?
• Emergency retrain on an NVSwitch rack fails until fabric manager matches the driver branch. What alignment fits?
• School-district dense box shows intermittent NCCL errors while fabric manager crash-loops overnight. What should ops stabilize?
• City monitoring alerts that fabric manager is inactive after image sync omitted enabling the unit. What should ops ensure?
• Assessor multi-GPU throughput halves while fabric manager reports partial NVSwitch links down. What should ops interpret?
• BCM head node disk fills with an old monitoring DB and municipal Base View freezes. What troubleshooting fits?
• A new water node provisions but never phones home because BCM network boot and DHCP settings are wrong. What should ops fix?
• City image sync reports success, but GPU nodes diverge after reboot. BCM software images do not match node categories. What should ops fix?
• After BCM head-node failover, Slurm job submits through BCM fail for the transit authority. What should ops repair?
• Public-health BCM alerts flood the NOC with non-actionable noise. What should ops tune?
• Transit operators cannot open Base View after the BCM GUI certificate expired. What fixes access?
• County BCM user authentication fails after directory sync; operators lose expected roles. What should ops repair?
• School-lab GPU nodes show as unknown in BCM after motherboard swaps. What reconciles identity?
• Emergency-management BCM admin commands hang cluster-wide. A stuck report job holds a database lock. What clears the freeze?
• City multi-GPU training stalls on IO. Magnum IO GPUDirect Storage to the parallel filesystem is misconfigured. What should ops fix?
• Public-health multi-node NCCL over InfiniBand is unexpectedly slow. What should ops verify in the Magnum IO stack?
• Transit checkpoint writes bypass GPUDirect and overload CPUs. What should ops check?
• Library CV containers show broken GPUDirect Storage after an image update. Magnum IO libraries no longer match. What restores GDS?
• Water-utility UCX and NCCL environment variables conflict with Magnum IO guidance after a toolkit bump. What should ops do?
• County storage disables a mount option required for GPUDirect Storage; Magnum IO falls back silently and training slows. What should ops recognize?
• Emergency multi-node jobs regress after an OFED update. What should ops consult before changing more packages?
• School-district single-node jobs are fine, but multi-node IO is broken. What Magnum IO area should ops troubleshoot?
• 311 feature-store ingest onto GPU nodes should use Magnum IO. Ops must prove the data path is healthy end to end. What fits?
• Assessor training GPUs sit idle while jobs wait on NFS metadata. What is the primary bottleneck to diagnose?
• Public-health checkpoints on spinning disks stall training for hours. What storage remediation fits?
• Transit parallel filesystem shows imbalance across OSTs and ETA training throughput drops. What restores balance?
• Library computer-vision datasets on object storage thrash with tiny reads. What improves the access pattern?
• Courts OCR batches compete with 311 embedding rebuilds on the same LUN and both slow down. What should ops apply?
• Water SCADA analytics nodes report GPUDirect Storage errors under load. What should ops validate?
• City NGC pulls fail authentication against the municipal mirror, blocking a Triton image deploy. What should ops fix?
• A public-health NGC container starts then exits because CUDA user-mode libraries are missing on the node. What closes the toolkit gap?
• Transit copied an NGC edge ARM tag onto x86 server nodes and the container will not run. What mismatch should ops catch?