What is a key advantage of the Transformer architecture over an RNN for natural language processing tasks?
Select an answer to reveal the explanation.
Short Explanation
Think about how an RNN has to work: word 1, then word 2, then word 3, one step at a time, because each hidden state depends on the one before it — you literally can't skip ahead. A Transformer breaks that chain. Thanks to self-attention, it can look at every word in the sequence at once and compute all those relationships in parallel, which is a massive speedup on modern hardware like GPUs, especially for long sequences. That's the headline advantage: it processes the whole sequence in parallel, speeding up training on long inputs. It's not that Transformers ignore word order — quite the opposite, they need positional encodings added explicitly since attention alone has no sense of order. It's not incompatible with attention — attention is literally the core mechanism it's built on. And it doesn't automatically mean fewer parameters; large Transformers are often bigger, not smaller, than comparable RNNs.
Full Explanation
One of the defining advantages of the Transformer architecture over recurrent neural networks for NLP tasks is its ability to process an entire input sequence in parallel rather than sequentially. In an RNN, computing the hidden state at time step t requires the hidden state from time step t-1, creating an inherently sequential dependency chain that prevents parallelization across the time dimension during training — this becomes a significant bottleneck for long sequences, both in terms of wall-clock training time and the difficulty of learning long-range dependencies due to vanishing gradients. The Transformer instead relies on self-attention mechanisms that compute relationships between all pairs of tokens in a sequence simultaneously, allowing the entire sequence to be processed in a single forward pass (per layer) rather than token by token; this parallelism maps extremely well onto modern GPU and TPU hardware, dramatically speeding up training especially for long inputs, and was a key motivation behind the original 'Attention Is All You Need' paper. The first distractor, requiring no positional information about word order, is incorrect and actually describes a limitation the Transformer must explicitly work around: because self-attention treats the input as an unordered set of tokens (it is permutation-invariant), Transformers must add positional encodings (sinusoidal or learned) to inject information about token order, which RNNs get 'for free' from their inherently sequential processing structure. The second distractor, being incompatible with attention mechanisms, is false on its face — attention (specifically multi-head self-attention) is the central, defining component of the Transformer architecture, not something it lacks or avoids. The third distractor, always using fewer total parameters than an equivalent RNN, is not a valid generalization; parameter count depends heavily on model design choices such as number of layers, hidden dimension size, and number of attention heads, and large-scale Transformer models (like modern LLMs) often have vastly more parameters than typical RNN-based models solving the same task, so parameter efficiency is not a guaranteed or defining property of the architecture. The parallelizability of self-attention computation, enabling much faster training on long sequences compared to the strictly sequential nature of RNNs, is the correct and most commonly cited practical advantage of Transformers in NLP.