How do you save and reload a trained Spark MLlib PipelineModel for later use in batch inference?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Spark MLlib models have their own native save/load API — model.save('/path') writes the model metadata and weights to distributed storage, and PipelineModel.load('/path') reads it back.
Full explanation below image
Full Explanation
Spark MLlib models implement save/load via the ML Persistence API: model.save('/dbfs/models/my_pipeline') writes the model to any path Spark can write to (DBFS, S3, ADLS, local). The model is stored as a directory containing: metadata/ (JSON with model parameters and class info), stages/ (subdirectory per pipeline stage). Reloading: from pyspark.ml import PipelineModel; loaded_model = PipelineModel.load('/dbfs/models/my_pipeline'). The loaded model can call loaded_model.transform(new_df) for batch inference. Why not pickle? Spark's distributed objects cannot be pickled — they contain JVM-side references. MLflow integration: mlflow.spark.log_model(model, 'spark_model') uses the native save/load under the hood and adds MLflow metadata. mlflow.spark.load_model(model_uri) reloads it. Both approaches work; MLflow integration adds versioning and lineage.