On a 2TB labeled dataset in Delta, which training approach is MORE…

Question

On a 2TB labeled dataset in Delta, which training approach is MORE practical?

Accepted Answer

Use distributed training approaches (e.g., Spark-compatible XGBoost/Spark ML / appropriate distributed frameworks) that scale with the cluster — 2TB → distributed training, not pandas-on-driver heroics.

Answer

Load all rows into pandas on the driver and run sklearn XGBoost

Answer

Ignore distributed compute and sample only 100 rows always

Answer

Email the dataset to a laptop

On a 2TB labeled dataset in Delta, which training approach is MORE practical?

Short Explanation and Infographic

Full Explanation