A batch inference job needs to apply a scikit-learn model (50MB) to billions of rows in a Spark DataFrame. What is the most efficient way to make the model available to all workers?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Broadcast variables ship the model from the driver to each worker exactly once, then cache it in memory — far better than having each of a billion tasks re-read from storage.
Full explanation below image
Full Explanation
spark.sparkContext.broadcast(model) serializes the model on the driver and sends it to each executor node exactly once, where it's cached in memory. All tasks on that executor share the same model object without re-reading it. Pattern: broadcast_model = spark.sparkContext.broadcast(loaded_model); then in a Pandas UDF: @pandas_udf('double'); def predict(features: pd.Series) -> pd.Series: model = broadcast_model.value; return pd.Series(model.predict(np.stack(features))); df.withColumn('prediction', predict('features_col')). Why not alternatives: loading from DBFS per task creates billions of file I/O operations (massive overhead). Storing as DataFrame column is impractical for model objects. mlflow.pyfunc.load_model() doesn't cache automatically per executor — it re-loads the model each time. mlflow.pyfunc.spark_udf() handles broadcasting automatically under the hood.