While training a vanilla RNN on long text sequences, an engineer notices the loss occasionally spikes to NaN and weight updates become huge and erratic. What is the standard, straightforward fix for this exploding-gradient problem?
Select an answer to reveal the explanation.
Short Explanation and Infographic
NaN losses and wild weight swings during RNN training are the classic signature of exploding gradients, and the standard fix is gradient clipping. You just cap the gradient's norm (or sometimes individual values) at some threshold before the optimizer applies the update, so no matter how large the raw gradient gets during backpropagation through time, the actual step taken stays bounded and sane. It's a quick, cheap, and remarkably effective patch. Increasing sequence length would make the problem worse, not better, since longer sequences mean more repeated multiplication through time, which is exactly what causes gradients to blow up. Switching to a feedforward network throws away the sequence modeling you need in the first place. And removing regularization has nothing to do with gradient magnitude during backprop, it addresses overfitting, a separate issue.
Full explanation below image
Full Explanation
Exploding gradients are a well-known instability that arises particularly in recurrent neural networks trained via backpropagation through time, where the same weight matrix is applied repeatedly across many time steps. If the largest eigenvalue of that recurrent weight matrix (or more generally, the relevant Jacobian) is greater than one, repeated multiplication across many time steps causes gradients to grow exponentially as they are backpropagated toward earlier steps, leading to numerical overflow, NaN losses, and wildly oscillating or diverging weight updates. Gradient clipping is the standard, practical remedy: before the optimizer applies the computed gradients to update the weights, the gradient vector's norm (or, less commonly, each individual gradient value) is checked against a chosen threshold, and if it exceeds that threshold, the gradient is rescaled down to have exactly that maximum norm, preserving its direction while bounding its magnitude. This simple technique prevents any single update from being catastrophically large, stabilizing training without requiring architectural changes, and is used as a standard component in training RNNs, LSTMs, GRUs, and even some transformer training regimes.
The distractor suggesting increasing sequence length is incorrect and in fact counterproductive: longer sequences mean gradients must be backpropagated through more time steps, which, in the presence of a Jacobian with eigenvalues greater than one, would exacerbate rather than mitigate the exponential growth responsible for the exploding-gradient problem. The distractor suggesting a switch to a plain feedforward network is incorrect because, while it's true that a feedforward network has no recurrent weight-sharing across time steps and therefore wouldn't experience this specific mode of gradient explosion, doing so abandons the sequential modeling capability that made an RNN necessary in the first place, and is not a targeted fix but rather an architectural surrender that changes the problem being solved. The distractor about removing the loss function's regularization term is incorrect because regularization terms like L2 weight decay address model capacity and overfitting, and have no direct causal relationship with the numerical instability of gradients growing exponentially during backpropagation through time; removing regularization would not address exploding gradients and could actually make overfitting worse.
A helpful memory aid: gradient clipping is like a circuit breaker, it doesn't prevent a power surge from occurring inside the wiring, but it does prevent that surge from frying your appliance, letting training continue safely even when individual gradient computations occasionally spike far higher than expected.