When sizing a Databricks cluster for Spark MLlib training on a 100GB dataset, what is the general recommended minimum RAM per executor?
Select an answer to reveal the explanation.
Short Explanation and Infographic
You don't need RAM equal to the full dataset — Spark processes it partition by partition. Size each executor to comfortably hold a few partitions (their working set) with room for shuffle buffers and JVM overhead.
Full explanation below image
Full Explanation
Spark processes data in partitions (default 128MB each). 100GB dataset ≈ 800 partitions at 128MB each. Sizing guidelines: 1) Partition processing — each executor task works on one partition at a time; executor RAM should hold 2-4 partitions comfortably: 128MB × 4 = 512MB per core minimum. 2) Shuffle memory — sorts and aggregations (like tree building in Random Forest) require additional memory for shuffle buffers. 3) JVM overhead — additional memory for JVM, serialization, Spark metadata. 4) ML working set — ML algorithms may need to hold sorted feature statistics in memory (for maxBins histograms). Practical rule: for ML workloads, 4-8GB per executor core is typical. Total cluster RAM goal: 3-4x the dataset size in RAM gives good performance without excessive disk spills. Caching (df.cache()) requires all data to fit in aggregate cluster memory, but processing (without caching) just needs enough RAM per partition batch.