When training a machine learning model using gradient descent, what is the primary consequence of setting the learning rate to an excessively small value?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: the learning rate is like the size of the steps you take when you're walking down a mountain in a thick fog. If you take giant leaps (a high learning rate), you might overshoot the valley floor and end up on the opposite ridge. But if you take tiny, microscopic steps (a small learning rate), you'll walk in a very straight and safe line, but it's going to take you three weeks to reach the bottom! In deep learning, an excessively small learning rate means your training is going to be incredibly slow. You'll burn through compute time and budget long before the model actually finishes learning. Go with Option D.
Full Explanation
In gradient descent optimization, the learning rate (often denoted as $\alpha$ or $\eta$) is a crucial hyperparameter that determines the step size taken in the direction of the negative gradient during each weight update.
When the learning rate is configured to be very small, the adjustment made to the model's weights in each iteration is minimal. While a small learning rate provides high precision and ensures that the optimizer is unlikely to overshoot the minimum or diverge, the primary trade-off is computational speed. The optimizer must take millions of tiny steps to make significant progress down the loss surface. Consequently, training requires many more iterations (epochs), leading to high wall-clock training times, increased server costs, and slower development cycles. In practice, if the learning rate is too small, the updates can become smaller than the machine's floating-point precision, causing the model to stall and stop improving altogether.
Let's evaluate the incorrect options: - Option A is incorrect because a small learning rate actually increases the likelihood of getting stuck in local minima (since it lacks the momentum to jump out of shallow valleys), but it does not guarantee doing so, and this is not the main operational disadvantage. - Option B is incorrect because a small learning rate will eventually minimize training error given sufficient time and iterations. - Option C is incorrect because overfitting is a function of model capacity, training duration, and lack of regularization, not the learning rate value itself.
For the exam, remember the core trade-off: a high learning rate causes instability and divergence, while a low learning rate leads to extremely slow training convergence.