Which Python package provides LightGBM integration optimized for distributed training on Databricks clusters?
Select an answer to reveal the explanation.
Short Explanation
The standard lightgbm package runs single-node. For distributed LightGBM across a Spark cluster, you need SynapseML (formerly MMLSpark) which provides LightGBMClassifier and LightGBMRegressor that integrate with Spark DataFrames.
Full Explanation
SynapseML (formerly Microsoft Machine Learning for Apache Spark / MMLSpark) provides com.microsoft.azure.synapse.ml.lightgbm (Scala/Java) and synapse.ml.lightgbm (Python) which include: LightGBMClassifier, LightGBMRegressor, LightGBMRanker — all following the Spark ML Pipeline Estimator/Transformer API. These distribute LightGBM training across all cluster workers using a network-based parallelism similar to LightGBM's native distributed mode. This enables training on datasets too large for a single node. The standard lightgbm Python package (pip install lightgbm) is single-node only. pyspark.ml.classification.LightGBMClassifier doesn't exist — LightGBM is not part of MLlib's built-in classifiers. databricks.lightgbm is a fabricated package name. SynapseML can be installed via Maven coordinates on Databricks clusters.