A team is training a sequence model on long text passages and finds that a standard RNN struggles to retain information from the beginning of the sequence by the time it reaches the end. Which architectural change most directly addresses this problem?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: a plain RNN keeps multiplying gradients as it backpropagates through time, and after enough steps those gradients shrink toward zero — that's the vanishing-gradient problem, and it's exactly why vanilla RNNs forget what happened early in a long sequence. The fix is the LSTM. It adds a memory cell plus input, forget, and output gates that decide what to keep, what to toss, and what to pass along, so useful signal can survive many time steps instead of decaying away. Bumping the learning rate doesn't touch the underlying gradient decay and can just make training unstable. More dense layers after the RNN don't help the RNN itself remember anything across time. And swapping softmax for sigmoid on the output is an activation choice, unrelated to how gradients flow backward through the sequence.
Full explanation below image
Full Explanation
Standard RNNs suffer from the vanishing-gradient problem: during backpropagation through time, gradients are repeatedly multiplied by the same weight matrix and activation derivatives at every time step. Over long sequences, this product shrinks exponentially toward zero, so the network effectively stops learning dependencies between distant time steps — early context gets lost by the time later tokens are processed. The LSTM (Long Short-Term Memory) architecture solves this directly by introducing a memory cell alongside three gates (input, forget, and output) that regulate how information flows through the cell state. Because the cell state update is largely additive rather than purely multiplicative through a nonlinearity at every step, gradients can flow across many time steps with far less decay, allowing the network to learn long-range dependencies. Increasing the learning rate is incorrect because it does not change the structural cause of vanishing gradients; it can even destabilize training or cause divergence. Adding more fully connected layers after the recurrent output increases model capacity for the final mapping but does nothing to change how gradients propagate through the recurrent time steps, so it does not help the network retain early-sequence information. Changing the output activation from softmax to sigmoid alters how outputs are interpreted (multi-class distribution versus independent probabilities) but has no bearing on gradient flow through recurrent connections over time. The LSTM's gating mechanism — and its close relative, the GRU — remains the standard architectural remedy for long-term dependency retention in sequence modeling, which is why it is a foundational building block before Transformers largely supplanted recurrent approaches for many NLP tasks.