When defining a Hyperopt search space, which distribution should you use for the learning rate parameter, and why?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Learning rate varies by orders of magnitude — 0.001 and 0.01 are very different, but 0.09 and 0.10 are nearly the same. Log-uniform ensures you sample as many values from 0.001-0.01 as from 0.01-0.1.
Full explanation below image
Full Explanation
hp.loguniform('lr', low, high) samples exp(uniform(low, high)), meaning the log of the value is uniformly distributed. For learning rate: hp.loguniform('lr', np.log(0.001), np.log(0.1)) samples values where log(lr) is uniform between log(0.001) and log(0.1), giving equal probability to each order of magnitude (0.001-0.01 and 0.01-0.1). hp.uniform('lr', 0.001, 0.1) would sample ~90% of values in the 0.01-0.1 range and only ~10% in 0.001-0.01, missing good learning rates in the lower range. Parameters appropriate for log-uniform: learning rate, regularization strength (C, alpha), and any parameter that spans orders of magnitude. Parameters appropriate for uniform: layer sizes, tree depth, dropout rates. hp.randint is for integer parameters. hp.normal can cause negative samples which are invalid for learning rate.