In Spark MLlib, what is the difference between an Estimator and a Transformer?
Select an answer to reveal the explanation.
Short Explanation and Infographic
An Estimator learns from data (like a student studying), and the result is a Transformer (the now-trained student who can answer questions without studying again).
Full explanation below image
Full Explanation
Spark MLlib's Pipeline API is built on two abstractions: Estimator — has a fit(DataFrame) method that learns parameters from data and returns a fitted Model (which is a Transformer). Examples: StandardScaler (learns mean/std), StringIndexer (learns category ordering), LogisticRegression (learns weights). Transformer — has a transform(DataFrame) method that applies a fixed transformation. Examples: StandardScalerModel (applies learned mean/std), StringIndexerModel (applies learned index), LogisticRegressionModel (applies learned weights for prediction). Pipeline itself is an Estimator — pipeline.fit(train_df) fits each stage in order and returns a PipelineModel (which is a Transformer) that can be applied with pipelineModel.transform(test_df). The distinction is fundamental: Estimators must never see test data during fit(); Transformers apply the same learned function everywhere.