Transit overnight ETA training must process each weekly dataset chunk. Which Slurm pattern should ops deploy?
Select an answer to reveal the explanation.
Short Explanation
Job arrays are the weekly route sheet—each dataset chunk gets a slot without someone typing sbatch at 2 a.m. Interactive shells and one catch-all job are how transit overnight trains go sideways.
Full Explanation
Use Slurm job arrays over weekly dataset chunks for transit overnight ETA training. Arrays fan out one template across partition indices (%a), so each weekly chunk becomes a tracked task with shared scripts, consistent GRES requests, and manageable dependencies—ideal for periodic civic retrains without manual 2 a.m. submissions.
One interactive shell left open overnight with manual restarts dies with disconnected sessions and has no fair accounting. A single job that ignores dataset partitions either OOMs or silently skips chunks. Separate human-submitted jobs typed by hand for every chunk at 2 a.m. do not scale and invite inconsistent flags across weeks.
Exam caveat: array size limits and MaxArraySize in Slurm matter; throttle with % concurrency limits so overnight arrays do not stampede the GPU partition. Use afterok dependencies if chunks must finish before a merge step. Make chunk indexing deterministic so failed tasks can be requeued without duplicating successful weeks.
Operational check: submit a small array over sample chunks, verify each task sees the right partition via SLURM_ARRAY_TASK_ID, confirm GPU GRES per task, and requeue a failed index only. Monitor queue time so Monday ETA serving still gets fresh weights.