What is learning-rate decay in the context of training a deep learning model?
Select an answer to reveal the explanation.
Short Explanation
Learning-rate decay is exactly what it sounds like: you start training with a reasonably large learning rate so the model can make big, fast progress early on, and then you gradually shrink that learning rate as training goes on. Early on, big steps get you into the right neighborhood fast. Later, small steps let you fine-tune and settle precisely near a good minimum instead of overshooting it. That's answer B. Randomly disabling neurons is dropout, a completely separate regularization idea. A penalty that shrinks weights over time describes weight decay or L2 regularization, not the learning rate. And stopping training once validation loss stalls is early stopping, another distinct technique. Learning-rate decay is purely about how big your steps are, not about killing neurons, penalizing weights, or ending training.
Full Explanation
Learning-rate decay refers to a scheduling strategy where the learning rate hyperparameter is systematically reduced over the course of training, following a schedule such as step decay (dropping the rate by a factor every N epochs), exponential decay, cosine annealing, or a 1/t decay curve. The rationale is that a larger learning rate early in training allows the optimizer to make rapid progress across the loss landscape, while a smaller learning rate later in training allows the model to take finer, more precise steps as it approaches a good minimum, reducing the risk of oscillating around or overshooting that minimum.
The first distractor describes dropout, a regularization technique that randomly zeroes out a fraction of neuron activations during training to prevent co-adaptation and reduce overfitting; it has no relationship to adjusting the learning rate over time. The second distractor describes weight decay (closely related to L2 regularization), which adds a penalty proportional to the squared magnitude of the weights directly into the loss function to discourage large weight values; this shrinks weights, not the learning rate itself, though the two techniques are sometimes confused because both involve a decaying quantity. The third distractor describes early stopping, a separate technique that halts training once a monitored metric (typically validation loss) stops improving, which is about when to stop training, not how the learning rate changes during training.
Common implementations include step decay, exponential decay, cosine annealing (which smoothly decreases the rate following a cosine curve, sometimes with warm restarts), and adaptive schedules tied to plateaus in validation performance (such as ReduceLROnPlateau). Learning-rate decay is widely used alongside adaptive optimizers like Adam, since even adaptive per-parameter scaling benefits from an overall reduction in step size as training converges.