A model's training loss barely moves epoch after epoch and stays stubbornly high. Weight updates are being computed correctly and gradients are non-zero. What is the most likely cause?
Select an answer to reveal the explanation.
Short Explanation
If gradients are flowing fine but your loss just refuses to budge, think about the size of the steps you're taking, not whether steps are happening at all. That's a learning rate that's too low — the model knows which direction to move in, it's just taking tiny, timid baby steps that barely dent the loss over many epochs. That's answer B. Too many training examples doesn't stall learning, it just means more data to learn from. Too many parameters relative to data describes overfitting risk, which shows up as a widening gap between train and validation loss, not a training loss that's flatlined high. And dropout set to zero just means no regularization is happening — that wouldn't stop the model from fitting the training data, if anything it'd make training loss drop faster. Bump the learning rate and watch that curve move.
Full Explanation
When gradients are correctly computed and non-zero but the loss curve stays flat and high across many epochs, the most common culprit is a learning rate that's too small. The learning rate scales how far each weight moves in the direction indicated by the gradient; if it's tiny, the optimizer takes minuscule steps and the loss decreases so slowly that it appears stuck, even though the model is technically still learning. Increasing the learning rate (or using a learning-rate finder / warmup schedule) typically resolves this immediately, causing the loss to drop noticeably faster.
The first distractor is wrong because dataset size does not cause a stalled loss curve — more examples simply means more signal to learn from, and while it can slow down each epoch computationally, it does not prevent the loss value itself from decreasing. The second distractor describes an overfitting-adjacent condition (excess capacity relative to data), but overfitting manifests as training loss continuing to fall while validation loss rises or plateaus — not as training loss staying flat and high, which is the scenario described here. The third distractor is wrong because a dropout probability of zero simply disables that regularization technique; it doesn't handicap the model's ability to reduce training loss, and in isolation would likely let training loss fall faster, not slower.
Diagnosing a stuck-high training loss should start with the learning rate: try a few orders of magnitude higher and watch whether the loss starts dropping. If it does, you've found the bottleneck. Other explanations exist (dead ReLUs, poor initialization, vanishing gradients in very deep networks), but a too-low learning rate is the single most common and easiest-to-fix cause of a training loss that plateaus early at a high value.