During training, the training loss keeps falling steadily, but the validation loss starts rising after a certain epoch. What does this pattern indicate?
Select an answer to reveal the explanation.
Short Explanation
This is the textbook signature of overfitting, plain and simple. Training loss keeps dropping because the model is getting better and better at memorizing the exact examples it's seeing over and over — quirks, noise, and all. But validation loss climbing back up tells you those memorized quirks don't generalize to data the model hasn't seen. The model's basically becoming a savant at the training set and worse at everything else. That's the correct answer. A too-high learning rate would usually show up as instability or oscillation in BOTH curves, not this specific fall-then-rise divergence pattern. Bad shuffling would show up as a weird or biased validation performance from the very start of training, not a gradual rise partway through. And batch size being too large is a hardware/memory concern, not something that produces this train/val divergence pattern. When you see the gap widen like this, think regularization, early stopping, or more data.
Full Explanation
The pattern described — training loss continuing to decrease while validation loss begins increasing after some point in training — is the classic signature of overfitting. As training continues past a certain point, the model starts to fit not just the underlying signal in the training data but also its noise and idiosyncrasies, effectively memorizing specifics of the training set that do not generalize to unseen data. This causes performance on the held-out validation set to degrade even as the model appears to be improving according to the training loss alone, creating a widening gap between the two curves.
If the learning rate is too high, training typically produces a loss curve that oscillates or fails to decrease smoothly, rather than the pattern of steadily falling training loss alongside rising validation loss; instability would be visible in the training curve, not exclusively the validation curve. If the training and validation sets were not shuffled before splitting, it would tend to produce biased validation performance from very early in training (e.g., if the validation set contains a different distribution), rather than a gradual divergence partway through training. Using a batch size that's too large is primarily a memory and hardware-capacity concern; it does not, by itself, cause the described train/validation loss divergence.
Common remedies for this overfitting pattern include early stopping (halting training at or restoring the model from the epoch where validation loss was lowest), adding or strengthening regularization (L1/L2 weight penalties, dropout), gathering more training data, reducing model capacity (fewer parameters or layers), and applying data augmentation to expose the model to more varied examples. Monitoring the gap between training and validation loss throughout training — rather than looking at either curve in isolation — is the standard diagnostic practice for catching overfitting as early as possible.