What is the main performance advantage of a Pandas UDF (vectorized UDF) over a standard Python UDF in Spark?
Select an answer to reveal the explanation.
Short Explanation and Infographic
A regular Python UDF hands each row across the Java-Python bridge one at a time. A Pandas UDF shoves the whole batch across at once using Arrow — like the difference between ferrying passengers one at a time vs. loading a whole bus.
Full explanation below image
Full Explanation
Standard Python UDFs serialize each row individually between the JVM (Spark's runtime) and the Python interpreter using pickle, which is extremely slow for large datasets. Pandas UDFs (decorated with @pandas_udf) use Apache Arrow for columnar, batch-oriented serialization — entire column batches are transferred at once, which is 10-100x faster. The function receives a pandas.Series (or pandas.DataFrame for grouped UDFs) and must return a pandas.Series. Pandas UDFs are categorized as: SCALAR (row-wise operation on Series), SCALAR_ITER (iterator of Series batches), GROUPED_MAP (split-apply-combine on DataFrame groups), GROUPED_AGG (custom aggregation). They still run in Python worker processes on executor nodes, not on the driver, and don't bypass the JVM — they still interact with Spark's scheduler.