When using standard Recurrent Neural Networks (RNNs) to analyze long sequences of text or time-series data, the model often struggles to retain information from the early parts of the sequence because gradients shrink to zero during backpropagation. Which architecture resolves this issue, and what is its primary benefit?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: standard RNNs have a short memory. If you give them a long paragraph of text, by the time they get to the end, they've completely forgotten what happened at the beginning. Mathematically, this is because of the vanishing gradient problem—as the gradients get passed back through time, they shrink to zero and learning stops. Enter the Long Short-Term Memory network, or LSTM. Think of an LSTM cell like a smart conveyor belt with gates. These gates—specifically the input, forget, and output gates—decide exactly what information to keep, what to throw away, and what to pass along. This clever design lets the network retain long-term dependencies across hundreds of time steps without the gradients vanishing. Yes, it's more complex and expensive to train than a standard RNN, but it's a lifesaver for sequence data!
Full explanation below image
Full Explanation
Recurrent Neural Networks (RNNs) are designed to process sequential data by maintaining a hidden state that acts as a memory of past inputs. However, standard RNNs suffer from a major limitation known as the vanishing gradient problem. During backpropagation through time (BPTT), gradients are multiplied repeatedly by the weight matrix at each time step. If these weights are small, the gradients shrink exponentially as they flow back to early time steps, preventing the model from updating its weights and learning long-term dependencies. To overcome this limitation, Hochreiter and Schmidhuber introduced the Long Short-Term Memory (LSTM) network. The key innovation of an LSTM is its cell state (the memory cell) and its unique gating mechanism. An LSTM cell contains three gates that control the flow of information: The Forget Gate determines what information from the previous cell state should be discarded; The Input Gate decides which new information should be stored in the cell state; The Output Gate controls what information from the cell state is output as the hidden state for the next time step. This structure allows the cell state to carry information over long sequences with minimal modification, effectively creating an uninterrupted path for gradients to flow backward. Consequently, LSTMs mitigate the vanishing gradient problem and successfully capture long-term dependencies in sequential data. Standard feedforward networks and CNNs are not inherently designed to maintain sequential memory over variable lengths, and autoencoders are primarily used for representation learning or dimensionality reduction, making them incorrect choices for this task.