For modern Spark ML feature engineering, prefer DataFrames/Spark ML…

Question

For modern Spark ML feature engineering, prefer DataFrames/Spark ML because:

Accepted Answer

DataFrame/Dataset APIs integrate with Catalyst/AQE and Spark ML Pipelines more cleanly than raw RDDs — Modern path: DataFrames + Spark ML + Catalyst — not raw RDD heroics by default.

Answer

RDDs are always faster with Catalyst

Answer

DataFrames cannot hold numbers

Answer

RDDs have better optimizer integration always

For modern Spark ML feature engineering, prefer DataFrames/Spark ML because:

Short Explanation and Infographic

Full Explanation