In Spark MLlib's DecisionTreeClassifier, what does the maxBins parameter control?
Select an answer to reveal the explanation.
Short Explanation and Infographic
maxBins is a discretization parameter — for continuous features, Spark bins the values into at most maxBins buckets and only considers bucket boundaries as split candidates. Higher maxBins = finer splits but more memory.
Full explanation below image
Full Explanation
Spark MLlib's distributed decision tree algorithm requires that all possible split candidates be determined at each node. For continuous features, Spark: 1) Computes the approximate quantiles of each feature across the dataset. 2) Creates at most maxBins - 1 threshold values (bins) as split candidates. 3) Finds the best split from these candidates. maxBins must be >= max number of categories in any categorical feature (or you get an error). Default: maxBins=32. Higher maxBins: more accurate splits (closer to finding the true optimal split), but requires more memory per partition to store the histogram. Lower maxBins: faster computation, less memory, but potentially missing the best split threshold. Important: maxBins does NOT control the maximum number of bins in the output features — it controls the number of candidate splits evaluated during tree construction. For categorical features with high cardinality, maxBins must be >= the number of unique values.