During training of a very deep network, gradients for the earliest layers become vanishingly small so those weights barely update. What problem is being described?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's where it bites people in production and on the exam. You stack a ton of layers, start training, and the early layers just sit there — frozen like they never got the memo. That's vanishing gradients: the error signal dies out as it travels backward. Think of it like a whisper passed through twenty people; by the time it reaches the first person, it's silence. Exam trap: don't pick exploding weights (that's the other disaster), "loss is zero," or "we never find any minimum." The signature is tiny gradients in the first layers. Mitigations you should know: ReLU-family activations, residual connections, careful init, normalization. Deep net, early layers not learning? Vanishing gradient.
Full explanation below image
Full Explanation
The vanishing gradient problem arises in deep neural networks when gradients of the loss with respect to parameters in early layers become extremely small during backpropagation. Because gradients are products of many Jacobian factors along the computational graph, saturating nonlinearities (classic sigmoid and tanh), poor initialization, or extreme depth can drive those products toward zero. Early weights then receive negligible updates while later layers may still learn, yielding ineffective depth and slow or stalled overall training.
The correct description matches this mechanism: backpropagated signals shrink so early layers barely move. Historical deep sigmoid networks suffered heavily; modern practice mitigates with ReLU and variants, residual and skip connections, batch or layer normalization, better initialization schemes, and architectures designed for gradient flow. Monitoring gradient norms by layer is a practical diagnostic when early blocks appear frozen.
A loss of zero is unrelated as a definition: vanishing gradients can coexist with high residual error when only part of the network is learning. Claiming the optimizer finds no minimum overstates the failure mode; partial learning often continues in upper layers. Exploding gradients are the complementary issue—multiplicative factors grow large, causing unstable updates and huge weights—and are addressed with clipping, careful scaling, and architectural choices, not by treating them as the same pathology as vanishing.
Underlying principle: deep credit assignment requires that error signals remain numerically useful as they travel backward. Best practice is to design for gradient flow from the start (residuals, appropriate activations, normalization) rather than only reacting after training fails. Memory aid: vanishing means the signal dies going backward and early layers are starved; exploding means the signal blows up and training becomes unstable. On exam questions, look for early layers, gradients near zero, and difficulty updating the first layers as the vanishing-gradient fingerprint.