For batch inference on a large Delta table using an MLflow-logged scikit-learn model, which approach is most efficient on Databricks?
Select an answer to reveal the explanation.
Short Explanation and Infographic
mlflow.pyfunc.spark_udf() is the bridge between your MLflow model and Spark — it wraps the model as a UDF so predictions run in parallel across all executors on your data, not bottlenecked on the driver.
Full explanation below image
Full Explanation
mlflow.pyfunc.spark_udf(spark, model_uri, result_type='double') creates a Spark UDF that: 1) Broadcasts the model to all executor nodes. 2) Applies predictions in parallel, partition by partition, leveraging all cluster cores. 3) Returns predictions as a new Spark column. Usage: predict_udf = mlflow.pyfunc.spark_udf(spark, 'models:/fraud_detector/3'); predictions = df.withColumn('prediction', predict_udf(*feature_cols)). Why not alternatives: df.toPandas() loads all data to driver memory — fails for large tables, single-threaded. REST API row-by-row creates massive network overhead (thousands of HTTP calls). Converting sklearn to Spark MLlib is complex and may change prediction behavior. This pattern scales to billions of rows.