Excessive HPO parallelism can hurt when:
Select an answer to reveal the explanation.
Short Explanation
Hyperopt's TPE is like a detective who gets smarter with every clue, so if you launch a huge batch of trials at once, most of them start before any clues come back. That's the parallelism vs. adaptivity trade-off: you finish faster, but each trial learns less from the ones before it, and all those concurrent workers can spike your cluster bill.
Full Explanation
Hyperopt proposes new trials based on the results of earlier ones, and Databricks documents an explicit trade-off between parallelism and adaptivity for SparkTrials. With parallelism set to 1, an adaptive algorithm such as Tree of Parzen Estimators sees every previous result before choosing the next point, so it can steer the search efficiently. As parallelism approaches max_evals, many trials are launched before any results return, so the search behaves more like random sampling and may need more evaluations to reach the same quality, while concurrent workers raise compute cost. Saying the harm arises when trials are free and infinite fails because free, unlimited trials would remove the cost pressure that makes lost sample efficiency matter. Claiming MLflow cannot log parallel runs fails because SparkTrials logs each trial as a child run under the parent run, including records written from workers. Saying clusters can never run more than one task fails because Spark runs many tasks concurrently, which is exactly how SparkTrials parallelizes trials; SparkTrials also caps parallelism at the number of concurrent tasks the cluster allows. Exam caveat: expect the phrase trade off scalability against adaptiveness. Operational check: compare best loss and total cluster time for the same max_evals at a low and a high parallelism setting.