Kernel timelines for clerk SFT show unfused LayerNorm dominating step time. Which change is most appropriate?
Select an answer to reveal the explanation.
Short Explanation
Lots of tiny LayerNorm launches are like stopping at every driveway on a clerk courier route. Fuse them into one efficient stop and the step timer drops without rewriting the whole SFT stack.
Full Explanation
Kernel timelines for clerk supervised fine-tuning that show unfused LayerNorm dominating step time usually indicate excessive kernel-launch overhead and repeated memory traffic for a numerically simple op. Switching to fused LayerNorm kernels collapses normalize-scale-bias work into fewer launches with better memory access, cutting step time without redesigning the rest of the SFT stack. Increasing host print logging inside every LayerNorm call adds synchronization and host overhead that make the timeline worse and distort measurements. Replacing LayerNorm with an unprofiled custom Python loop on CPU moves a hot path onto the slowest execution environment and destroys GPU pipeline efficiency. Disabling all fused kernels solely to make timelines easier to read prioritizes pretty profiles over clerk training throughput and reintroduces the launch tax the fusion was meant to remove. Exam caveat: fusion gains assume the framework’s fused op matches the training precision and layout in use; mismatched dtype or an unexpected residual pattern can force a fallback to unfused paths that looks like a “fusion” regression. Operational check: after enabling fused LayerNorm, compare Nsight Compute or framework profiler step breakdowns and confirm LayerNorm’s share of step time drops while loss curves for the clerk corpus remain within the prior validation band.