A team switches from a simple vanilla RNN to an LSTM for a language modeling task involving paragraphs hundreds of words long. What is the main advantage this switch provides?
Select an answer to reveal the explanation.
Short Explanation
The whole reason LSTMs exist is to fix a specific weakness in vanilla RNNs: they struggle to remember anything from far back in a long sequence because gradients shrink toward zero as they're backpropagated through many time steps, the classic vanishing gradient problem. LSTMs fix this with a dedicated memory cell and a set of gates, input, forget, and output, that let the network decide what information to keep, discard, or pass along at each step, largely independent of how many steps have passed. That's what lets an LSTM connect something mentioned at the start of a long paragraph to something at the end. LSTMs don't skip backpropagation, they're trained with it just like vanilla RNNs. They typically have more parameters than a vanilla RNN of comparable hidden size, not fewer, because of the extra gates. And they don't run faster per step either, the gating mechanisms add computation, not remove it.
Full Explanation
Long Short-Term Memory (LSTM) networks were specifically designed to address the vanishing gradient problem that severely limits vanilla recurrent neural networks' ability to learn long-range dependencies. In a vanilla RNN, gradients must be backpropagated through time by repeatedly multiplying by the same recurrent weight matrix (and passing through repeated nonlinear activations) at every time step; when the relevant eigenvalues are less than one, this repeated multiplication causes gradients to shrink exponentially as they propagate backward through many steps, meaning that errors from far in the past have almost no influence on updates to weights relevant to early inputs. This makes it very difficult for a vanilla RNN to learn relationships between elements that are separated by long distances in a sequence, such as connecting a subject at the start of a long paragraph to a pronoun near its end. LSTMs address this by introducing a dedicated cell state that runs through the sequence with only minor, carefully regulated linear interactions at each step, combined with three gating mechanisms: an input gate controlling what new information is written into the cell state, a forget gate controlling what existing information is discarded, and an output gate controlling what part of the cell state is exposed as the hidden state output. Because the cell state can be preserved largely unchanged across many time steps when the forget gate is close to one, gradients can flow backward through time with much less attenuation, allowing LSTMs to learn dependencies over sequences that are substantially longer than what vanilla RNNs can effectively handle.
The distractor about eliminating backpropagation is incorrect because LSTMs are trained using the same backpropagation through time procedure as vanilla RNNs; they do not replace gradient-based learning with any fixed or rule-based feature extraction mechanism. The distractor claiming LSTMs always use fewer parameters is incorrect and actually backwards: LSTMs introduce three separate gates plus the cell-state pathway, each with its own weight matrices, meaning an LSTM layer has roughly four times the parameters of a vanilla RNN layer with the same hidden dimension, not fewer. The distractor about running faster per time step by skipping activation functions is incorrect because LSTMs still apply nonlinear activations (sigmoid gates and tanh transformations) at each step, and the additional gating computations make an LSTM more computationally expensive per time step than a vanilla RNN, not less.
A helpful memory aid: think of the LSTM's cell state as a conveyor belt running alongside the main recurrent computation, information can ride along that belt largely undisturbed unless a gate specifically decides to add, remove, or read from it, which is precisely what allows information (and gradients) to survive much longer journeys through time than in a vanilla RNN.