When preparing data with missing values for a Spark MLlib pipeline, which approach allows the Imputer transformer to handle multiple columns at once?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Spark's Imputer is vectorized — you hand it a list of input columns and a matching list of output columns, and it fills gaps in all of them at once.
Full explanation below image
Full Explanation
pyspark.ml.feature.Imputer supports vectorized multi-column imputation via the inputCols and outputCols parameters. This is essential for building proper ML pipelines because: 1) It fits the imputation strategy (mean, median, mode) on training data and applies the same learned values to test/serving data — preventing data leakage from computing statistics on the full dataset. 2) It integrates seamlessly into a Pipeline object alongside other transformers and the final estimator. 3) Setting outputCols to new names preserves the original columns for inspection. df.fillna() is a DataFrame method that computes statistics on the entire dataset including test data (leakage risk) and doesn't integrate into a pipeline. SQLTransformer with COALESCE requires manually providing the fill value, not learning it from data. Imputer does support multiple columns — the 'one column only' option is incorrect.