Traditional Recurrent Neural Networks (RNNs) struggle to maintain information when processing long sequences, such as long paragraphs of text. What major mathematical limitation of standard RNNs led to the development of Long Short-Term Memory (LSTM) networks?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: imagine you're reading a book, and by the time you reach page fifty, you've completely forgotten what happened in chapter one. That's a standard RNN for you! During backpropagation through time, the gradients get multiplied over and over. If those gradients are decimal values less than one, they shrink to basically zero. This is the vanishing gradient problem. The model loses its ability to update weights for older inputs, meaning it has zero long-term memory. LSTMs fixed this with their 'cell state'—think of it like a memory highway that lets older information flow straight through without getting crushed. This is a big exam topic, so lock it in!
Full Explanation
Standard Recurrent Neural Networks (RNNs) process sequence data step-by-step, maintaining a hidden state that is updated at each time step. However, when training on long sequences, standard RNNs suffer severely from the vanishing gradient problem during Backpropagation Through Time (BPTT). Because the same weight matrix is multiplied repeatedly across many time steps, if the eigenvalues of the weight matrix are less than one, the gradients will shrink exponentially. By the time the gradient propagates back to the earliest steps, it becomes infinitesimally small, meaning the weights corresponding to early steps receive no meaningful updates. As a result, the network cannot learn long-term dependencies. Long Short-Term Memory (LSTM) networks were specifically designed to solve this issue. They introduce a cell state (acting as an information conveyor belt) and gating mechanisms (forget, input, and output gates) that regulate the flow of information. The gates allow gradients to flow backwards through time with minimal decay, preserving long-term dependencies. Standard RNNs are not inherently unable to handle varying lengths, they do not lack loss functions, and they are actually computationally lighter (with fewer parameters) than LSTMs, making the other options incorrect.