What design feature allows Long Short-Term Memory (LSTM) networks to successfully capture long-range temporal dependencies compared to standard Recurrent Neural Networks (RNNs)?
Select an answer to reveal the explanation.
Short Explanation
Remember how standard RNNs have terrible memory because of vanishing gradients? LSTMs are the upgrade. Think of an LSTM like a smart conveyor belt (the cell state) running through the network. Along this belt, we have three gates: the forget gate (which dumps useless information), the input gate (which adds new relevant info), and the output gate (which decides what to show next). Because information can flow down the conveyor belt with very little change, the gradients don't vanish during training. This lets LSTMs remember things from hundreds of steps ago. That's why Option C is correct!
Full Explanation
Standard Recurrent Neural Networks (RNNs) suffer from vanishing gradients when backpropagating through long sequences, which prevents them from learning long-term dependencies. Long Short-Term Memory (LSTM) networks, introduced by Hochreiter & Schmidhuber, solve this problem by redesigning the recurrent unit structure. The core of an LSTM is the cell state (often described as a conveyor belt or memory cell), which runs straight down the entire chain of units with only minor linear interactions. The flow of information to and from the cell state is regulated by three neural network layers called gates: 1) Forget Gate: Decides what information from the previous cell state should be discarded using a sigmoid activation function. 2) Input Gate: Decides which new information from the current input should be stored in the cell state. 3) Output Gate: Decides what the next hidden state should be based on the updated cell state. Because the cell state update involves linear addition rather than repeated matrix multiplication, gradients can flow backward through time without vanishing, allowing LSTMs to effectively learn long-term sequences. Let's review the other options: Option A is incorrect because LSTMs are specifically designed to handle long sequences, not restrict them. Option B is incorrect because LSTMs are recurrent architectures, not pure feedforward networks. Option D is incorrect because LSTMs are sequence models (used for text, audio, etc.), not specialized computer vision networks. For the exam, memorize the three gates (forget, input, output) and the cell state as the defining components of an LSTM that resolve the vanishing gradient issue.