A model's loss oscillates wildly and never settles down, even growing over time. Gradients are correct. What hyperparameter is most likely set too high?
Select an answer to reveal the explanation.
Short Explanation
Wild oscillation and a loss that actually grows over time instead of settling down is the classic fingerprint of a learning rate set too high. The model keeps overshooting the minimum on every single update — like slamming on the gas and blowing straight past your parking spot every time you try to pull in. That's the correct answer. Dropout rate being high would hurt final accuracy and make training noisier in a different way, but it wouldn't cause this kind of runaway divergence. Too large a batch size actually tends to make the gradient estimate MORE stable, not less, so that's not your culprit either. And too many epochs just means training runs longer — it doesn't, by itself, cause the loss to diverge. When you see this bouncing, exploding pattern, drop the learning rate first.
Full Explanation
When the learning rate is set too high, each weight update overshoots the direction indicated by the gradient by such a large margin that the model repeatedly jumps past the minimum of the loss surface instead of converging toward it. This produces the classic symptom described: the loss oscillates violently from step to step and, in severe cases, can grow without bound (diverge) rather than decrease, because each update pushes the weights further from a good solution instead of closer to it.
Setting the dropout rate too high controls how many neuron activations are randomly zeroed during training; while this degrades capacity and increases noise, it does not typically cause the loss to diverge in the runaway fashion described. Using a larger batch size actually produces a more stable, less noisy gradient, the opposite of causing wild oscillation; while an excessively small batch size can add noise, the described symptom of unbounded growth is a hallmark of learning rate, not batch size. Using too many training epochs controls how many passes over the data; while this risks overfitting, not the specific oscillating-and-diverging behavior within the loss curve.
The standard remedy is to reduce the learning rate, often by an order of magnitude, and observe whether the loss begins to decrease smoothly. Additional mitigations include gradient clipping (capping the gradient norm to prevent any single update from being too large), using a learning-rate warmup phase to ease into training gradually, or switching to an adaptive optimizer like Adam, which can partially compensate for a suboptimal fixed learning rate by adjusting per-parameter step sizes based on gradient history.