What is the vanishing gradient problem, and in which setting is it especially severe?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: during backprop, gradients have to travel backward through a chain of multiplications. If those factors are small, the signal dies out before it reaches the early layers—like shouting down a long hallway and getting silence at the far end. That's vanishing gradients. Pay close attention here, because people mix this up with exploding gradients, where the numbers blow up instead. Vanishing gradients are a famous pain in deep RNNs unrolled over long sequences: early time steps barely get a learning signal. The exam loves that combo—what it is plus where it hurts most. Modern fixes include careful initialization, ReLU-family activations, residual connections, and gated RNNs like LSTM/GRU. Remember: shrink through depth/time = vanishing; deep RNNs are ground zero.
Full explanation below image
Full Explanation
The vanishing gradient problem arises when partial derivatives of the loss with respect to early parameters become extremely small as errors are backpropagated through many multiplicative steps. In deep feedforward stacks, each layer contributes a Jacobian; products of many factors with magnitudes less than one drive gradients toward zero. In recurrent networks, the same dynamics appear across time: unrolling the recurrence creates a deep computational graph, so long-range temporal dependencies may receive negligible gradient. Early weights then update too little, training stalls for those layers, and the model fails to capture long-horizon structure even if capacity exists on paper.
Deep recurrent architectures are a canonical setting where this concern is acute. Standard tanh or sigmoid recurrent units repeatedly squash signals, and long sequences multiply that effect. That is why the correct answer pairs exponential gradient decay during backpropagation with deep RNNs as a primary worry. Exploding gradients—the opposite pathology—occur when products amplify, causing unstable updates; clipping and careful design address that case separately.
Distractors often redefine the problem incorrectly. Claiming gradients become too large describes explosion, not vanishing. Saying the loss approaches zero confuses optimization success (or memorization) with gradient magnitude through depth. Asserting that the model simply has too few parameters or that the issue is unique to convolutional filters misses the mechanism: it is about derivative flow along long chains, not raw parameter count or a single architecture family. CNNs can also suffer depth-related gradient issues, but textbooks and exams historically emphasize deep RNNs for sequential vanishing gradients.
Practical mitigations include gated cells (LSTM, GRU) that create freer gradient paths, residual and dense skip connections, normalization layers, ReLU-style activations that avoid saturating everywhere, and optimizers with adaptive step sizes. Transformers further reduce pure sequential depth by using attention over full contexts. For exam success, define vanishing gradients as exponentially decaying backpropagated signals that impede early-layer learning, and associate the classic severity case with deep recurrent unrolling over long sequences.