During training of a recurrent neural network on long sequences, the loss suddenly spikes to NaN and the weight values become enormous. What phenomenon is most likely responsible?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Sudden NaNs and huge weights during RNN training are the classic signature of exploding gradients. Here's why it happens: an RNN reuses the same weight matrix at every time step, so when you backpropagate through time, the gradient gets multiplied by that same matrix over and over. If those repeated multiplications push values above 1, the gradient balloons exponentially with sequence length, and the weight updates blow up the whole network. A too-small learning rate would slow things down, not blow them up. Full convergence with overfitting looks like a low, stable training loss, not a NaN spike. And a fully saturated activation would tend to kill gradients (vanish them), not explode them — that's the opposite failure mode.
Full explanation below image
Full Explanation
The exploding gradient problem occurs in recurrent neural networks because the same weight matrix is applied repeatedly across time steps during backpropagation through time (BPTT). If the largest eigenvalue of that recurrent weight matrix is greater than 1, the gradient magnitude compounds multiplicatively across time steps, growing exponentially with sequence length. This leads to enormous weight updates, numerical overflow, and the NaN losses and runaway weight values described in the scenario. Common mitigations include gradient clipping (capping the gradient norm at a threshold), careful weight initialization, and using gated architectures like LSTM or GRU that are more resistant to this instability.
The first distractor is wrong because a learning rate that is too small produces slow, stagnant training — the loss would decrease very gradually or plateau, not spike to NaN or produce enormous weights; that symptom is the opposite of what a tiny learning rate causes. The third distractor describes overfitting, which shows up as a low, stable (or still slowly decreasing) training loss alongside a rising validation loss — a numerically stable pattern, not the catastrophic instability of NaNs and exploding weights. The fourth distractor describes gradient saturation, most associated with sigmoid or tanh activations flattening at their extremes; saturation actually causes gradients to shrink toward zero (contributing to the vanishing gradient problem), which is the mirror-image failure of what's described here, not the cause of exploding values.
Memory aid: vanishing gradients make training grind to a halt (numbers shrink toward zero across many time steps), while exploding gradients make training blow up (numbers grow without bound) — both stem from repeatedly multiplying by the same recurrent weight matrix across a long sequence, just in opposite directions depending on whether that matrix's dominant eigenvalue is below or above 1.