In Hyperopt, what algorithm does tpe.suggest implement and why is it preferred over random search for hyperparameter tuning?
Select an answer to reveal the explanation.
Short Explanation
TPE stands for Tree-structured Parzen Estimator — it's a Bayesian optimizer that learns from each trial's result and smartly picks the next hyperparameter configuration to try, rather than sampling at random.
Full Explanation
TPE (Tree-structured Parzen Estimator) is a sequential model-based optimization (SMBO) algorithm that: 1) Builds a probabilistic model of which hyperparameter configurations lead to good results. 2) Divides observed configurations into 'good' (top γ% by metric) and 'bad' (remaining). 3) Fits separate density models l(x) over good and p(x) over bad configurations. 4) Selects next configuration by maximizing the Expected Improvement ratio l(x)/p(x). This means TPE focuses search on regions where good results have been found, unlike random search (rand.suggest) which is memoryless. Benefits vs random: fewer trials needed to find good hyperparameters (typically 2-5x more efficient). In fmin(fn=objective, space=space, algo=tpe.suggest, max_evals=100): after ~10 initial random trials, TPE uses results to guide subsequent sampling. For SparkTrials with parallelism, TPE is adapted to suggest batches of configurations accounting for in-flight trials.