Transit fine-tuning on A100s needs higher tokens/sec. Which optimization should be enabled?
Select an answer to reveal the explanation.
Short Explanation
Flip on automatic mixed precision and let the A100 Tensor Cores chew transit tokens faster. FP64-everywhere or turning Tensor Cores off is the opposite of a speed win.
Full Explanation
Transit fine-tuning on A100s that needs higher tokens/sec should enable automatic mixed precision so Tensor Cores accelerate matmuls in FP16 or BF16 while sensitive reductions stay in higher precision as the AMP recipe dictates. That is the standard throughput lever on Ampere-class GPUs for transformer training. Forcing FP64 everywhere typically underuses Tensor Cores, increases memory traffic, and slows training—the opposite of a tokens/sec optimization for transit models. Disabling Tensor Cores so kernels stay on pure CUDA cores intentionally leaves performance on the table and contradicts the hardware’s design for mixed-precision GEMMs. Training exclusively on CPU to avoid precision choices abandons the A100 investment and collapses tokens/sec far below any acceptable civic training SLA. Exam caveat: AMP is the expected path to engage Tensor Cores for LLM and transformer training throughput; “higher precision everywhere” is a stability or legacy choice, not a speed strategy on A100s. Operational check: enable AMP (or the framework’s equivalent autocast path), confirm Tensor Core activity in Nsight or profiler counters, compare tokens/sec before and after on the same transit corpus and batch size, watch for loss instability (prefer BF16 if FP16 spikes), and keep a reproducible seed and config checked into the municipal training repo.