Assessor imagery training must avoid MIG-sharing interference. How should the Slurm workload request capacity?
Select an answer to reveal the explanation.
Short Explanation
Exclusive nodes give assessor vision training a quiet gym—no MIG roommates thrashing weights. Crowded MIG slices and login-node CPUs are the opposite of isolation.
Full Explanation
Request exclusive nodes via Slurm so assessor imagery training avoids MIG-sharing interference. Exclusive allocation (--exclusive or partition/QoS policy) gives the job whole nodes—full GPUs without noisy MIG neighbors contending for memory bandwidth or compute. That isolation matters for throughput-sensitive municipal vision training.
Forcing the job onto heavily shared MIG slices with noisy neighbors reintroduces the interference exclusivity was meant to prevent. Pinning training to oversubscribed login nodes with interactive users harms both the trainer and everyone compiling on the login pool—login nodes are not batch accelerators. Disabling GPU GRES and training on random CPUs in the login pool abandons the hardware the imagery model needs.
Exam caveat: exclusive nodes cost utilization—use them when interference is proven or SLAs demand it, not for every tiny debug run. If the fleet is MIG-mode only, exclusivity may still share physical GPUs as MIG instances unless you also schedule a full-GPU partition; align partition GPU mode with the request. Respect fairshare so one assessor job cannot permanently warehouse nodes.
Operational check: submit with exclusive flags to the correct partition, confirm scontrol show job reports whole-node allocation and expected full GPUs, compare step times against a MIG-shared baseline, and release nodes promptly after training.