What is the main downside of setting the learning rate far too low when training a neural network?
Select an answer to reveal the explanation.
Short Explanation
Picture walking toward a destination in steps so tiny you barely move — that's a learning rate set way too low. You're going in the right direction, technically, but training drags on forever, and because your steps are so small, you're much more likely to get trapped in some shallow dip or flat plateau in the loss landscape and never have enough momentum to climb back out. That's the real downside, answer A. Exploding gradients are actually more of a high-learning-rate or deep-network-instability problem, the opposite scenario. Overfitting instantly isn't caused by a low learning rate — if anything, a tiny learning rate means the model is barely learning at all, let alone overfitting fast. And batch normalization keeps doing its job regardless of what learning rate you pick; it's not tied to that hyperparameter.
Full Explanation
An excessively low learning rate causes the optimizer to take very small steps in parameter space during each update. This has two compounding consequences: training takes far longer to reach a good solution because the loss decreases extremely slowly, and the small step size makes it easier for the optimizer to get trapped in local minima, saddle points, or flat plateaus in the loss surface, since it lacks the momentum or step magnitude needed to escape shallow regions and continue toward a better minimum.
The first distractor, exploding gradients, is typically associated with the opposite problem — a learning rate that is too high, or with poorly initialized deep/recurrent networks where repeated multiplication of large gradient values causes numerical instability; a very low learning rate does not cause this. The second distractor is incorrect because overfitting requires the model to actually fit the training data well (and then some), which happens over many effective training steps; a learning rate so low that training crawls will typically underfit rather than instantly overfit, since the model hasn't had the chance to fit much of anything yet. The third distractor is incorrect because batch normalization's internal computation (normalizing layer activations using batch statistics, then scaling and shifting with learned parameters) operates independently of the learning rate value chosen for the overall optimizer; a poor learning rate can slow how well those learned scale/shift parameters converge, but it doesn't break batch norm's mechanism.
In practice, learning rate is one of the most sensitive hyperparameters in deep learning, and picking a value that's too low is a common early-training mistake. Techniques like a learning-rate range test, warmup schedules, and adaptive optimizers (Adam, RMSprop) exist precisely to help find and maintain a learning rate that is neither so high it destabilizes training nor so low it stalls progress.