When building natural language processing pipelines, developers frequently choose Transformer-based models over Recurrent Neural Networks (RNNs). What architectural difference primarily explains this preference?
Select an answer to reveal the explanation.
Short Explanation
You pick Transformers because they process the whole sequence at once with self-attention, so you can train faster on GPUs and keep long-range context. RNNs crawl word by word and don't hold onto early parts well. On the exam, remember: RNNs are sequential and forgetful; Transformers are parallel and attentive!
Full Explanation
The transition from Recurrent Neural Networks (RNNs) to Transformer architectures represents a paradigm shift in natural language processing (NLP). The primary limitation of RNNs (including LSTMs and GRUs) is their sequential nature. To process the t-th word in a sequence, an RNN must first compute the hidden states for all preceding steps 1 to t-1. This sequential dependency prevents parallelization during training, restricting the use of modern multi-GPU hardware. Furthermore, as the sequence length increases, the gradients tend to vanish or explode, making it difficult for the network to retain information from early time steps (the long-range dependency problem).
In contrast, the Transformer architecture discards recurrence entirely in favor of the self-attention mechanism. This design allows the model to ingest the entire sequence of tokens simultaneously. The self-attention mechanism calculates a set of attention weights for each word relative to every other word in the sequence, regardless of their distance. This allows the model to capture semantic relations across long spans of text in a single step. Because the computations for all tokens are independent of prior states, they can be processed in parallel across GPUs, drastically accelerating training efficiency on massive datasets. That is why the correct explanation is the one stating that Transformers process entire sequences simultaneously through self-attention, enabling parallel computation and better retention of long-range context compared to step-by-step RNN processing.
The claim that Transformers store sequence history in a static internal memory buffer that does not need to be updated during backpropagation is wrong. Transformers do not rely on a static buffer; their parameters are still trained through backpropagation, and their advantage comes from self-attention over the whole sequence. In practice, Transformers excel at handling long sequences, whereas RNNs struggle with them.
The claim that RNNs require labeled dataset pre-training, whereas Transformers can perform zero-shot learning on raw data without any training phase, is also wrong. Transformers require training data just like RNNs; they cannot perform deep language tasks without training. They are also more complex to design and implement than standard RNNs because of their multi-head attention and positional encoding systems.
The claim that Transformers rely on convolutional filters that eliminate the need to calculate positional relationships between words in a sentence is wrong as well. Transformers do not use convolutional filters as their core mechanism; they use self-attention and positional information to model relationships among tokens. On the exam, remember: RNNs are sequential and forgetful; Transformers are parallel and attentive!