During the training of a deep neural network, you monitor the training logs and notice that the loss value is fluctuating violently from epoch to epoch. Over the next few iterations, the loss explodes and eventually prints as 'NaN' or infinity. What is the most probable configuration issue causing this behavior?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Imagine your boss walks in and asks why the new image classifier is spitting out 'NaN' or infinity for its loss values after just a few epochs. You look at the logs and see the loss jumping from 2.0 to 10.0, then 50.0, and then boom—infinity. Think of the learning rate as the size of the steps you take when trying to find the bottom of a deep valley in the dark. If you take small, careful steps, you'll get there eventually. But if your learning rate is too high, it's like wearing giant seven-league boots. You take a step, leap completely over the bottom of the valley, and land high up on the opposite wall. On the next step, you leap even further back up the other side! Before you know it, you're bouncing higher and higher out of the valley until you launch into outer space. That's exactly what's happening to your optimizer—it's overshooting the minimum. Trust me on this, if you ever see loss oscillating wildly and exploding, the very first knob you need to turn down is your learning rate!
Full explanation below image
Full Explanation
The learning rate is one of the most critical hyperparameters in training neural networks. It determines the size of the step the optimization algorithm takes at each iteration when updating the model's weights in the direction of the negative gradient.
If the learning rate is configured incorrectly, it can severely disrupt the training process: 1. Learning Rate Too High (Divergence): When the learning rate ($\eta$) is set to an excessively large value, the weight updates are too large. Instead of descending the loss gradient toward the minimum, the optimizer overshoots. This can cause the loss to oscillate with increasing amplitude. If the steps continue to grow, the network's weights and activations will explode, leading to numerical instability and resulting in loss values of 'NaN' or infinity. 2. Learning Rate Too Low (Stagnation): Conversely, if the learning rate is too small, the optimizer will take tiny steps. While this makes convergence stable, the model will train extremely slowly, potentially getting stuck in local minima or taking an impractical amount of time to reach the global minimum.
Let's evaluate the incorrect options: - The dataset is too small: A small dataset typically causes overfitting (where the model memorizes the training data and performs poorly on validation data). It does not cause the training loss to oscillate to infinity. - The model is under-equipped (underfitting): An underparameterized model will fail to capture the patterns in the data, resulting in a high training loss that remains flat and refuses to decrease. It does not cause the loss to explode to infinity. - The learning rate is too low: As described, a low learning rate causes slow, smooth convergence, not wild oscillations or divergence.