Which major limitation of standard Recurrent Neural Networks (RNNs) are Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks specifically engineered to overcome?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal: standard RNNs forget early steps because gradients shrink during backpropagation through time. LSTMs and GRUs add gates that let information flow through, so you can learn long-range dependencies.
Full Explanation
Recurrent Neural Networks are designed to process sequential data by maintaining a hidden state that carries information from previous steps. However, training standard RNNs on long sequences using backpropagation through time introduces a fundamental mathematical challenge: the vanishing, or disappearing, gradient problem. Because the network repeatedly multiplies weight matrices across many time steps, the gradients calculated for early time steps decay exponentially. Consequently, the network cannot update its weights effectively for early inputs, rendering it incapable of learning long-term dependencies. Long Short-Term Memory networks and Gated Recurrent Units were specifically designed to overcome this limitation. Long Short-Term Memory networks introduce a unique component called the cell state along with three regulatory mechanisms known as gates: the forget gate, the input gate, and the output gate. The cell state acts as an internal conveyor belt, allowing information to bypass the standard recurrent transformations. This linear pathway prevents gradients from vanishing as they flow back through time, enabling the network to learn relationships between elements separated by long intervals. Gated Recurrent Units achieve a similar effect with a slightly simplified architecture using update and reset gates. The high computational cost associated with parallel training on multi-GPU setups is wrong because Long Short-Term Memory networks and Gated Recurrent Units are actually more computationally expensive than standard RNNs and still suffer from sequential processing bottlenecks, unlike Transformers, which parallelize computations. The tendency of the network to overfit on small sequential datasets is wrong because overfitting is typically caused by model complexity, meaning too many parameters relative to the training data, rather than sequence length. The model's inability to handle input sequences of varying lengths is wrong because RNN architectures, including standard RNNs, Long Short-Term Memory networks, and Gated Recurrent Units, are naturally capable of handling input sequences of varying lengths through padding or dynamic recurrent execution.