You're monitoring the training of a new deep neural network on a GPU cluster, and you notice the loss curve is completely flat—basically a straight line from epoch one. You check your configurations and see the learning rate is set to an extremely tiny number, like 0.000000001. Why is this tiny learning rate preventing the model from learning?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: think of learning rate like the stride of your footsteps when you're hiking down into a valley (the minimum loss). If your stride is a tiny fraction of a millimeter, you're going to take forever to get down there. You'll look like you're standing completely still! That's exactly what's happening when your learning rate is too low. The weight updates are so microscopic that the loss curve looks like a flat line. If it was too high, you'd be taking giant leaps and jumping clean over the valley to the next mountain. We want that sweet spot, and this is a classic gotcha on the exam and in production!
Full explanation below image
Full Explanation
The learning rate is a critical hyperparameter that scales the magnitude of the updates applied to the model's weights during backpropagation. If the learning rate is set too low, the step size taken by the optimization algorithm (such as Stochastic Gradient Descent) is extremely small. Consequently, the weight updates are negligible, and the loss function will change so slowly that it appears as a flat line. This leads to extremely slow convergence or premature stagnation, where the model fails to find the local or global minimum within a reasonable number of epochs. Conversely, if the learning rate is set too high, the updates will be too large. This causes the optimizer to overshoot the minimum, leading to divergence or wild oscillations in the loss value (which would appear as a jagged or rising line, not a flat line). Overfitting occurs when a model performs exceptionally well on training data but poorly on unseen data; it does not present as a flat loss line from the start of training. Finally, optimizer complexity itself does not cause a flat line unless it interacts with a poorly tuned learning rate. Finding the right learning rate is typically achieved using learning rate finders or schedulers.