Profiling a city LLM shows Softmax+MatMul dominating step time. What attention-path action fits?
Select an answer to reveal the explanation.
Short Explanation
If Softmax and MatMul own the timeline, fuse the attention path so the city trainer stops paying for separate kernel launches. A CPU Python loop is not the performance answer.
Full Explanation
When profiling a city LLM shows Softmax and MatMul dominating step time, the fitting attention-path action is to adopt efficient fused attention kernels that collapse Softmax and MatMul traffic into a memory-efficient implementation and cut step time on the civic trainer. The profile already localized the hotspot; the remedy should target that path. Ignoring GEMM timelines and only retuning the learning rate does not address the computational bottleneck the profiler identified and wastes experiment budget. Replacing attention with a CPU Python loop for clarity collapses throughput and is pedagogical, not operational, for city-scale training. Increasing sequence length further without kernel changes increases attention cost quadratically in naive implementations and worsens Softmax+MatMul dominance. Exam caveat: profile-guided optimization targets dominant kernels—when attention GEMMs dominate, fix the attention implementation before unrelated hyperparameters or longer contexts. Operational check: enable fused or Flash-style attention in the training stack, re-profile to confirm Softmax+MatMul time drops, validate civic evaluation quality is unchanged, and document the kernel flag in the city training standard so later jobs inherit the win.