Domain 4: Distributed ML with Spark

Databricks Certified Machine Learning Associate · 49 questions

• In Spark MLlib, what is the difference between an Estimator and a Transformer?
• Why does Spark MLlib require all features to be assembled into a single Vector column before training?
• Which Spark MLlib class implements Gradient Boosted Trees for classification tasks?
• Before training an ML model with Spark MLlib, a large training DataFrame has very uneven partition sizes (some 50MB, others 500MB). What operation should be applied to optimize training performance?
• The Pandas Function API (applyInPandas) is used for which ML use case?
• What is the purpose of Horovod in distributed deep learning on Databricks?
• Using ParamGridBuilder in Spark MLlib, how many total models will be trained if you add 3 values for maxDepth and 4 values for numTrees to a RandomForestClassifier?
• How do you save and reload a trained Spark MLlib PipelineModel for later use in batch inference?
• A batch inference job needs to apply a scikit-learn model (50MB) to billions of rows in a Spark DataFrame. What is the most efficient way to make the model available to all workers?
• When a Spark MLlib Pipeline includes both a StringIndexer and a LogisticRegression, what exactly is saved when you call pipelinemodel.save(path)?
• How can a trained Spark MLlib PipelineModel be applied to a Structured Streaming DataFrame for real-time batch scoring?
• In Spark MLlib's DecisionTreeClassifier, what does the maxBins parameter control?
• What is the purpose of Pandas on Spark (formerly Koalas) in a Databricks ML workflow?
• What problem does Petastorm solve when training deep learning models on large Spark Delta Lake datasets?
• During a custom distributed training loop using Spark, how can you count the number of rows that failed validation across all executors?
• When sizing a Databricks cluster for Spark MLlib training on a 100GB dataset, what is the general recommended minimum RAM per executor?
• Best stage order for a Spark ML text classification Pipeline?
• VectorAssembler's role in Spark ML is to:
• When is Spark MLlib generally preferable to single-node sklearn on Databricks?
• Horovod on Databricks is MOST relevant for:
• Before distributed training on a skewed dataset, repartitioning can help by:
• Vectorized pandas UDFs (Iterator/Series) are preferred over row-at-a-time Python UDFs because:
• coalesce(n) vs repartition(n) — which is accurate?
• Saving a fitted Spark ML PipelineModel enables:
• Broadcast joining a small dimension table of features to a large fact table is good when:
• Petastorm is used to:
• Increasing maxDepth on a Spark DecisionTreeClassifier typically:
• Spark accumulators are appropriate for:
• Pandas API on Spark (formerly Koalas) helps ML feature engineering by:
• In Spark ML, stages share a features column name convention so that:
• A practical bottleneck in multi-node DL training can be:
• In Spark ML, the difference between Estimator and Transformer is:
• applyInPandas / mapInPandas group-wise is useful when:
• Training thousands of per-SKU models is often implemented via:
• Persisting intermediate feature DataFrames helps when:
• OOM on executors during vectorized inference often fixed by:
• VectorSlicer is used to:
• Low-level mapPartitions inference is sometimes used to:
• Adaptive Query Execution (AQE) can help ML feature jobs by:
• Oversizing all-purpose clusters for light interactive EDA wastes:
• Spark ML ALS is primarily for:
• Checkpointing during long DL training enables:
• IndexToString is used to:
• Spark whole-stage codegen benefits feature ETL by:
• Using partition columns as raw features can be problematic when:
• Data locality awareness helps distributed ML by:
• Enabling Arrow for pandas UDFs primarily improves:
• For scheduled training jobs, prefer job clusters because:
• For modern Spark ML feature engineering, prefer DataFrames/Spark ML because: