When a Spark MLlib Pipeline includes both a StringIndexer and a LogisticRegression, what exactly is saved when you call pipeline_model.save(path)?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Saving a PipelineModel saves everything — each fitted stage's learned state: the vocabulary dictionary that StringIndexer built, the weights that LogisticRegression learned, all of it.
Full explanation below image
Full Explanation
When pipeline.fit(train_df) produces a PipelineModel, all stages are fitted: StringIndexerModel (learned vocabulary: {'cat': 0, 'dog': 1, 'bird': 2}), LogisticRegressionModel (coefficients, intercept). Calling pipeline_model.save('/dbfs/models/my_pipeline') saves: A metadata directory with JSON files describing the class name and hyperparameters of each stage. A stages directory with one subdirectory per stage, each containing the stage's model data (the vocabulary for StringIndexerModel, coefficients for LogisticRegressionModel, etc.). Loading: PipelineModel.load('/dbfs/models/my_pipeline') reconstructs all stages with their fitted states. This is a full round-trip — the loaded model behaves identically to the original. It's NOT a pickle — Spark uses a custom JSON+Parquet format that's robust to Python version changes and is readable by Scala/Java Spark code too.