When processing sequential data with a standard Recurrent Neural Network (RNN), which of the following represents the primary training limitation that limits its ability to learn relationships across long sequences?
Select an answer to reveal the explanation.
Short Explanation
Think of a standard RNN like a person reading a massive novel with memory that lasts only a few sentences. When backpropagation through time runs backward, gradients get multiplied over and over, and small gradients shrink to nothing. That leaves no signal to adjust early weights, so you lose long-term dependencies.
Full Explanation
Recurrent Neural Networks (RNNs) are designed to process sequential data by maintaining an internal hidden state that acts as memory. However, when trained using Backpropagation Through Time over long sequences, standard RNNs encounter a severe limitation known as the vanishing gradient problem. During the backward pass, the gradient of the loss function is propagated back through the temporal steps. This involves repeated matrix multiplications of the network's recurrent weight matrices. If the eigenvalues of these matrices are less than one, the gradients shrink exponentially as they propagate backward across many time steps. Consequently, the weight updates for the early steps in the sequence become negligible, meaning the network fails to learn long-term dependencies, or relationships between inputs separated by large temporal distances. The correct choice captures this: it struggles to maintain long-term dependencies due to the vanishing gradient problem during backpropagation through time. The claim that it is completely unable to handle variable-length sequential inputs is incorrect because standard RNNs are conceptually designed to process sequences of variable lengths. The claim that it requires pre-computed spatial feature maps from a convolutional network to function is incorrect because RNNs do not require spatial feature maps from convolutional networks; they can process raw sequential vectors, such as text tokens or sensor data, directly. The claim that it exhibits extremely high computational efficiency that causes the network to overfit instantly is incorrect because RNNs are actually computationally intensive and prone to slow sequential execution, which does not inherently cause immediate overfitting. For the exam, remember that vanishing gradients are the core reason standard RNNs struggle with long-term dependencies, prompting the development of LSTMs and GRUs.