Courts OCR model training crawls when the Slurm working directory sits on slow NFS. What placement fix fits?
Select an answer to reveal the explanation.
Short Explanation
Slow NFS is molasses under the courts OCR kitchen—move the Slurm cwd to fast scratch so epochs cook. USB sticks and WAN-only I/O are how municipal trains stall.
Full Explanation
Relocating the Slurm training working directory to fast civic scratch storage fixes the courts OCR slowdown because training I/O—checkpoints, scratch tensors, shuffle buffers, and epoch intermediates—is latency- and bandwidth-sensitive. Slow citywide NFS can serve durable archives, but it saturates under concurrent write storms typical of GPU training. Placing the Slurm cwd on local or cluster scratch that matches GPU throughput lets municipal OCR epochs progress without waiting on shared-filer contention, while still allowing operators to stage finished artifacts to NFS when the job completes.
Keeping checkpoints and scratch forever on the slowest citywide NFS share fails by storage-tier mismatch: durable NFS is for persistence and sharing, not as the hot working set for every training write. A USB stick taped to a courtroom podium fails by media and reliability concept—removable consumer storage lacks cluster durability, bandwidth, multi-node visibility, and operational governance. Disabling local scratch and forcing every epoch across the WAN to a remote county fails by network-path concept: WAN round-trip and shared remote bandwidth amplify the same I/O stall the courts already see on NFS.
Exam caveat: the correct fix is placement of the working directory and scratch, not a claim that NFS must never store models. Municipal policy may still require final checkpoints on governed NFS or object storage after training; the anti-pattern is making that slow tier the live cwd during active epochs.
Operational check: compare epoch step time and checkpoint write duration with the cwd on NFS versus civic scratch; confirm Slurm job scripts and #SBATCH or --chdir settings point at the fast path, and verify node-local or parallel scratch capacity and cleanup policies before the next OCR retrain.