Transit ops configures NeMo burn-in for HGX-class nodes with finite GPU memory. What sizing rule applies?
Select an answer to reveal the explanation.
Short Explanation
You’ll get a meaningful burn-in by choosing a representative NeMo configuration that fits the GPU memory on your target nodes. Oversizing it just to look tough can turn the test into a crash loop instead of a useful health check.
Full Explanation
Transit ops should select a representative NeMo burn-in configuration sized for available GPU memory on the target HGX-class nodes. That lets the run exercise realistic AI-framework paths and intended municipal workloads without invalid out-of-memory storms caused by reckless oversizing. Domain 4 framework burn-in is meaningful only when the configuration fits the accelerators under test and remains representative. Choosing the largest possible model regardless of GPU memory intentionally creates allocation failures, obscuring whether a node is healthy or simply mis-sized for the test. Ignoring memory and relying on host swap for all model weights causes pathological performance and does not represent how GPU training should run on HGX nodes. Deploying an existing profile from another GPU generation without checking memory needs or compatibility can mismatch tensor cores, memory capacity, or driver features, producing misleading pass/fail outcomes. Exam caveat: “representative” and “fits memory” are joint constraints—a toy configuration can falsely pass a weak node, while an oversized one can falsely fail a healthy node; use a documented profile for the GPU SKU. Practical check: inventory GPU memory on the transit HGX nodes, choose the approved NeMo burn-in profile for that SKU, verify the run stays within device memory without relying on host swap, and record the configuration ID with the results.