A common pattern for batch inference at scale on Databricks is:

Question

Accepted Answer

Distribute model inference with Spark (pandas UDFs / mlflow.pyfunc.sparkudf) over a DataFrame — Batch score with Spark UDFs/pandas UDFs — not a driver for-loop over billions of rows.

Answer

One-at-a-time REST calls from a for-loop on the driver for billions of rows

Answer

Manual copy-paste predictions

Answer

Retraining every row

A common pattern for batch inference at scale on Databricks is:

Short Explanation and Infographic

Full Explanation