In Spark MLlib's DecisionTreeClassifier, what does the maxBins parameter control?
Select an answer to reveal the explanation.
Short Explanation
maxBins is a discretization parameter — for continuous features, Spark bins the values into at most maxBins buckets and only considers bucket boundaries as split candidates. Higher maxBins = finer splits but more memory.
Full Explanation
Spark MLlib's distributed decision tree algorithm requires that all possible split candidates be determined at each node. For continuous features, Spark: 1) Computes the approximate quantiles of each feature across the dataset. 2) Creates at most maxBins - 1 threshold values (bins) as split candidates. 3) Finds the best split from these candidates. maxBins must be >= max number of categories in any categorical feature (or you get an error). Default: maxBins=32. Higher maxBins: more accurate splits (closer to finding the true optimal split), but requires more memory per partition to store the histogram. Lower maxBins: faster computation, less memory, but potentially missing the best split threshold. Important: maxBins does NOT control the maximum number of bins in the output features — it controls the number of candidate splits evaluated during tree construction. For categorical features with high cardinality, maxBins must be >= the number of unique values.