A researcher plots the training loss curve for a new model and sees it spike up and down wildly from epoch to epoch instead of trending downward, even though the architecture and data pipeline are known to be correct. What is the most likely explanation?
Select an answer to reveal the explanation.
Short Explanation
Picture gradient descent as trying to walk down into a valley. If your steps (the learning rate) are way too big, you don't gently descend — you overshoot the bottom, bounce to the other wall, overshoot again, and your loss zig-zags or even explodes instead of smoothly dropping. That's exactly what erratic, spiky training loss tells you: the learning rate is too high, answer B. A too-large batch size (option A) usually makes the loss curve smoother and more stable, not less — it's actually the opposite direction of the problem you're describing. Too few layers (option C) would show up as an underfitting model with a loss that plateaus too high, not one that's bouncing around erratically. And the validation set size (option D) has nothing to do with training loss instability — that's a training-loop symptom, not a data-split issue.
Full Explanation
When a training loss curve oscillates wildly instead of decreasing smoothly, the most common and most likely cause is a learning rate that is set too high. Gradient descent updates weights by moving them a step in the direction opposite the gradient, and the learning rate scales the size of that step. If the step is too large, each update can overshoot the minimum of the loss surface, landing the weights on the far side of a valley rather than settling into it. Successive updates then bounce back and forth across the minimum, or across steep regions of the loss landscape, producing the large, erratic swings seen in the loss curve. In severe cases, an excessively high learning rate can cause the loss to diverge entirely, growing without bound.
A batch size that is too large is incorrect as the explanation because larger batches produce gradient estimates that are closer to the true gradient over the full dataset (lower variance), which tends to produce a smoother loss curve, not a more erratic one. If anything, a very small batch size introduces more per-step noise, but that noise is generally mild compared to the drastic instability caused by an oversized learning rate, and it is not the standard explanation for wild, erratic spikes.
An insufficient number of layers is incorrect because model capacity issues (a network too shallow or too small to fit the data) typically manifest as underfitting: training loss plateaus at a relatively high value and stops improving, rather than swinging unpredictably epoch to epoch.
An undersized validation set is incorrect because validation set size affects the reliability of the validation metric, not the training loss itself; it is unrelated to why the training loss curve would be unstable.
Memory aid: 'erratic loss = step too big.' The standard fix is to reduce the learning rate (or add a learning-rate schedule / warmup, or use gradient clipping) so that each update takes a smaller, more controlled step toward the minimum.