A researcher trains a plain (vanilla) RNN on sequences several hundred time steps long and finds it struggles to learn dependencies between early and late elements in the sequence. What is the primary architectural reason for this limitation?
Select an answer to reveal the explanation.
Short Explanation and Infographic
This is the textbook long-sequence problem with plain RNNs: vanishing and exploding gradients. Because the same weight matrix gets applied over and over as you backpropagate through time, the gradient signal either shrinks toward zero or blows up exponentially the further back you go, so information (and the learning signal that would encode it) from early time steps barely reaches the parameters that need updating. That's why a vanilla RNN forgets what happened hundreds of steps ago. It's not that RNNs can't run in parallel (true, but that's a speed issue, not a long-range-memory issue), there's no hard-coded length limit baked into the math, and just adding more hidden units per layer doesn't fix a gradient-flow problem — it might even make it worse.
Full explanation below image
Full Explanation
The correct answer is the vanishing/exploding gradient problem. Vanilla RNNs backpropagate error signals through time by repeatedly multiplying by the same recurrent weight matrix (and the derivative of the activation function) at every time step. Across hundreds of steps, this repeated multiplication compounds: if the relevant values are consistently less than 1, gradients shrink exponentially toward zero (vanishing), and the network effectively cannot learn dependencies between distant time steps because the learning signal from far-back inputs never meaningfully reaches earlier weights. If the values are consistently greater than 1, gradients grow exponentially (exploding), causing unstable, erratic updates. Either way, the network fails to capture long-range dependencies — exactly the symptom described. This is the core motivation behind gated architectures like LSTM and GRU, which use gating mechanisms to preserve gradient flow across long sequences.
The first distractor mislabels the issue as being about parallelism: it's true that RNNs process sequences step-by-step and cannot be parallelized across the time dimension the way Transformers can, but that is a computational-efficiency limitation, not the reason for failing to learn long-range dependencies — a slow-but-parallelizable RNN would still suffer from vanishing gradients if it existed. The second distractor invents a fictional constraint; there is no hard-coded maximum sequence length in the RNN's mathematical formulation — RNNs can technically process sequences of any length, they simply become increasingly ineffective at retaining information as sequences grow longer. The third distractor confuses model capacity with gradient flow; adding more hidden units increases the dimensionality of the hidden state and the network's representational capacity per time step, but it does not address the multiplicative gradient decay/growth across time steps, so it does not solve the long-range dependency problem.
Memory aid: 'RNNs have a short memory because the message gets weaker (or wilder) every time it's passed down a long chain' — that's vanishing/exploding gradients in a sentence.