What does the exploding gradient problem describe in deep network training?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Pay close attention—this one bites people in production and on tests. Exploding gradients means the error signal blows up as it flows backward. Weights get slammed with huge updates, loss spikes, NaNs show up, training goes off the rails. Think of turning the volume knob past eleven until the speakers shred. Vanishing gradients are the opposite movie: signal fades to nothing. Don't confuse explosions with "learning rate too low" or "loss went nicely to zero." Fixes you'll actually use: gradient clipping, better init, residual paths, gated RNNs, careful learning rates. If your boss asks why the run diverged at step 200, exploding grads are on the suspect list. Remember: explode equals too big; vanish equals too small. You've got this.
Full explanation below image
Full Explanation
The exploding gradient problem occurs when partial derivatives of the loss with respect to parameters become extremely large during backpropagation. In deep feedforward stacks or long unrolled recurrent computations, repeated multiplication by large Jacobian singular values amplifies the error signal. The optimizer then applies oversized updates, causing parameters to jump far from useful regions of the loss surface. Observable symptoms include sudden loss spikes, oscillating metrics, numerical overflow, and NaN values in activations or weights.
This failure mode is complementary to vanishing gradients, where signals shrink and early layers barely move. Both arise from the chain rule through deep computational graphs, but they require different mental models: explosion is instability from amplification; vanishing is stalled learning from attenuation. Architectures and practices that improve gradient flow—residual connections, careful initialization (for example, He/Xavier variants), normalized layers, gated recurrent units, and attention-based alternatives to very deep recurrence—reduce both risks, though not identically.
Mitigations specifically aimed at explosions include gradient clipping (by value or global norm), lower or scheduled learning rates, mixed-precision loss scaling handled correctly, and detecting divergence early in training logs. Simply declaring that the learning rate is "too low" does not define the problem; an overly high learning rate can worsen explosions but is a related hyperparameter issue, not the gradient pathology itself. Likewise, a loss approaching zero describes successful optimization, not gradient explosion.
For exams, lock the definitions: exploding = gradients become excessively large and destabilize updates; vanishing = gradients disappear and hinder learning. Knowing both terms and at least one practical countermeasure (especially clipping) is a high-yield combination for deep learning fundamentals sections.