When using scikit-learn's Pipeline with cross_val_score() on Databricks, what is the key benefit of wrapping preprocessing and the estimator in a Pipeline?
Select an answer to reveal the explanation.
Short Explanation and Infographic
A scikit-learn Pipeline inside cross-validation is the correctness guarantee — without it, your scaler 'sees' the validation fold during fit, poisoning your evaluation. The Pipeline makes the right thing automatic.
Full explanation below image
Full Explanation
cross_val_score(pipeline, X, y, cv=5) with a Pipeline object is fundamentally different from cross_val_score(model, X_scaled, y, cv=5) (where X_scaled was computed before splitting). With a Pipeline: for each fold, cross_val_score calls pipeline.fit(X_train_fold, y_train_fold), which calls each transformer's fit_transform on training data only, then calls pipeline.predict(X_val_fold), which applies transform (not fit_transform) using the train-fold-fitted parameters. This guarantees no information from the validation fold leaks into the transformer fitting. scikit-learn Pipelines do NOT automatically distribute across Spark workers — they run on a single node. For distributed scikit-learn cross-validation on Databricks, libraries like joblib with Spark backend or hyperopt are needed.