What is the role of SparkTrials when using Hyperopt for hyperparameter tuning on Databricks?
Select an answer to reveal the explanation.
Short Explanation and Infographic
SparkTrials is the secret ingredient that turns Hyperopt from a sequential optimizer into a parallel cluster job — each trial (parameter configuration) runs on a separate worker simultaneously.
Full explanation below image
Full Explanation
Hyperopt's default Trials object runs trials sequentially on the driver node. SparkTrials (from hyperopt import SparkTrials) changes this behavior: fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=SparkTrials(parallelism=8)) distributes up to 'parallelism' trials simultaneously across Spark executor cores. Each trial runs the objective function (which typically trains a model) on a separate Spark task. This provides near-linear speedup for parallelism-count trials (8x speedup with parallelism=8 if you have 8 cores). Important: SparkTrials logs each trial as a separate MLflow child run under a parent run automatically. SparkTrials is NOT required — you can use Trials for sequential search. It does not replace the objective function or auto-store to Delta.