Nsight Systems shows the city’s 311 trainer idle on GPU while CPU tokenization saturates. What fix matches the profile?
Select an answer to reveal the explanation.
Short Explanation
When the GPU is twiddling its thumbs while the CPU chews tokens, you can see that more silicon won’t fix the bottleneck. Prefetching or async workers stage batches ahead of time—like permits before the counter opens—so your 311 training can keep moving.
Full Explanation
When Nsight Systems shows the city’s 311 trainer idle on GPU while CPU tokenization saturates, the accelerator is starved by the data pipeline rather than undersized. Moving tokenization and batch preparation off the critical path with prefetch queues or asynchronous workers keeps the next microbatch ready, allowing Tensor Cores to stay fed across steps. Buying more GPUs without changing the data pipeline simply adds idle silicon waiting on the same host tokenization bottleneck. Lowering Tensor Core utilization so the GPU waits longer for tokens accepts the stall instead of removing it and wastes training capacity. Disabling CUDA profiling forever makes stalls disappear from reports, not from execution, and removes the diagnostic signal needed to assess performance. Exam caveat: prefetching helps only when worker count, pinned memory, and storage I/O keep pace; if the corpus is on a contended NAS, GPU idle time may persist until I/O or decode concurrency improves. Practical check: after enabling async workers, rerun Nsight Systems on a 311 epoch slice and confirm GPU utilization rises while host tokenization appears on overlapping worker timelines rather than blocking the training step start.