A practical bottleneck in multi-node DL training can be:
Select an answer to reveal the explanation.
Short Explanation
Multi-node DL often bottlenecked by network sync and slow workers.
Full Explanation
Scale-out is not free; communications matter.