Before training an ML model with Spark MLlib, a large training DataFrame has very uneven partition sizes (some 50MB, others 500MB). What operation should be applied to optimize training performance?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Uneven partitions cause stragglers — one slow worker with a massive partition holds up the whole job. repartition() does a full shuffle to create equal-sized partitions, so all workers finish at the same time.
Full explanation below image
Full Explanation
Skewed partitions (some very small, some very large) cause performance problems in ML training: large partitions create straggler tasks that delay the entire training stage. df.repartition(n) performs a full shuffle to create n approximately equal-sized partitions — ideal for rebalancing skewed data before ML training. df.coalesce(n) reduces partition count without a shuffle by merging adjacent partitions — useful for writing fewer files but doesn't fix skew (large partitions remain large). df.cache() stores the DataFrame in memory but doesn't fix partition skew. Sorting is generally harmful before ML training (it creates skew by putting similar examples near each other). Best practice for ML: target partition sizes of 128-256MB, ensure count of partitions ≥ 2 × number of cores.