A practical bottleneck in multi-node DL training can be:
Select an answer to reveal the explanation.
Short Explanation and Infographic
Multi-node DL often bottlenecked by network sync and slow workers.
Full explanation below image
Full Explanation
Scale-out is not free; communications matter.