What is the purpose of Horovod in distributed deep learning on Databricks?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Horovod is the MPI-inspired glue that lets TensorFlow and PyTorch models train across all your GPUs at once — each GPU trains on its shard of data, then gradients are averaged across all workers via all-reduce.
Full explanation below image
Full Explanation
Horovod (open-sourced by Uber) implements data-parallel distributed training using the ring all-reduce algorithm: 1) Each worker (GPU/process) gets a shard of training data. 2) Each worker computes gradients on its local batch. 3) All-reduce aggregates (averages) gradients across all workers — each worker receives the same averaged gradient. 4) All workers update their model weights identically. This achieves near-linear scaling: 8 GPUs ≈ 8x faster training. On Databricks: HorovodRunner provides native Spark integration: hr = HorovodRunner(np=8); hr.run(train_fn). The train function uses hvd.init(), hvd.rank() to identify worker, and wraps the optimizer with hvd.DistributedOptimizer(). Horovod works with both TensorFlow and PyTorch. It is not Databricks-specific — it's an open-source framework that runs on any cluster.