In deep learning, what does hyperparameter tuning mean?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Let's clear a classic mix-up. Weights are what the model learns during training. Hyperparameters are the dials you set first—learning rate, batch size, number of layers, dropout, and friends. Tuning is the hunt for good dial settings: grid search, random search, Bayesian optimization, whatever your stack uses. If your boss says “loss is exploding,” you might lower the learning rate—that's a hyperparameter move, not the same as one SGD step. Trap: people call gradient updates “hyperparameter tuning.” Nope. Preprocessing isn't tuning either. Land it: tune the knobs before/around training; learn the weights during training.
Full explanation below image
Full Explanation
Deep learning systems distinguish parameters from hyperparameters. Parameters are the weights and biases adjusted by the optimizer to minimize the training loss. Hyperparameters are configuration choices that shape the model and learning process but are not typically updated by ordinary backpropagation: learning rate and schedules, optimizer choice and momentum, batch size, network depth and width, kernel sizes, dropout rates, weight decay, early-stopping patience, and similar controls. Confusing these two layers of configuration is one of the most common vocabulary mistakes on machine-learning exams and in early project planning conversations.
Hyperparameter tuning is the systematic (or heuristic) search for hyperparameter values that yield strong validation performance. Common strategies include manual trial-and-error, grid search, random search, Bayesian optimization, bandit-based methods, and population-based training. Because each trial may require a full or partial training run, teams budget compute carefully and use validation metrics—not training loss alone—to decide winners and reduce overfitting to the test set. Automated tools (for example, Optuna, Ray Tune, or cloud hyperparameter services) still implement this outer-loop idea: propose settings, train, score, and propose again.
Distractors target nearby vocabulary. Selecting among already trained models is model selection or checkpoint selection; it may follow tuning but does not define the search over training knobs. Preprocessing transforms data; while pipeline choices can themselves be tuned, “hyperparameter tuning” in standard deep-learning terminology centers on model and training configuration, not the general definition of cleaning inputs. Backpropagation-driven weight updates are parameter optimization—the inner loop—whereas hyperparameter tuning is the outer loop that decides how that inner loop should run.
Practical tips: tune learning rate early because it often dominates outcomes; change one factor at a time when debugging mysterious regressions; log every trial with seeds and data versions; use nested validation when comparing many configurations; and freeze a final hyperparameter set before final test evaluation. Exam phrase to remember: hyperparameters are set before training; tuning finds their best values, while gradient descent learns the weights.