In Hyperopt, what algorithm does tpe.suggest implement and why is it preferred over random search for hyperparameter tuning?
Select an answer to reveal the explanation.
Short Explanation and Infographic
TPE stands for Tree-structured Parzen Estimator — it's a Bayesian optimizer that learns from each trial's result and smartly picks the next hyperparameter configuration to try, rather than sampling at random.
Full explanation below image
Full Explanation
TPE (Tree-structured Parzen Estimator) is a sequential model-based optimization (SMBO) algorithm that: 1) Builds a probabilistic model of which hyperparameter configurations lead to good results. 2) Divides observed configurations into 'good' (top γ% by metric) and 'bad' (remaining). 3) Fits separate density models l(x) over good and p(x) over bad configurations. 4) Selects next configuration by maximizing the Expected Improvement ratio l(x)/p(x). This means TPE focuses search on regions where good results have been found, unlike random search (rand.suggest) which is memoryless. Benefits vs random: fewer trials needed to find good hyperparameters (typically 2-5x more efficient). In fmin(fn=objective, space=space, algo=tpe.suggest, max_evals=100): after ~10 initial random trials, TPE uses results to guide subsequent sampling. For SparkTrials with parallelism, TPE is adapted to suggest batches of configurations accounting for in-flight trials.