In Spark MLlib, which class implements k-fold cross-validation for hyperparameter tuning?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Spark MLlib's cross-validation class is called CrossValidator — similar to scikit-learn's GridSearchCV in purpose but a Spark-native class that parallelizes fold training across the cluster.
Full explanation below image
Full Explanation
pyspark.ml.tuning.CrossValidator implements k-fold cross-validation in Spark MLlib. It requires: estimator (the ML pipeline or model), estimatorParamMaps (a ParamGrid — created with ParamGridBuilder().addGrid().build()), evaluator (e.g., BinaryClassificationEvaluator), and numFolds (default 3). CrossValidator trains the estimator k times, each time using (k-1) folds for training and 1 fold for evaluation, averaging the metric across folds. The best hyperparameter combination is used to retrain on the full dataset. Important: CrossValidator parallelizes across fold-parameter combinations using Spark, making it much faster than scikit-learn's sequential approach on large datasets. GridSearchCV is scikit-learn's class. KFoldValidator and ParamGridSearch don't exist in either framework.