What major capability made the Transformer architecture the dominant foundation for modern natural language processing (NLP) models compared to traditional Recurrent Neural Networks (RNNs)?

The Transformer architecture, introduced by Vaswani et al. in 2017, replaced recurrent neural networks (RNNs, LSTMs, GRUs) as the state-of-the-art framework for sequential data. Its primary breakthrough is the self-attention mechanism, which allows the model to process all tokens in a sequence in parallel rather than sequentially. This parallelization eliminates the bottleneck of sequential step-by-step processing and enables the network to capture dependencies and relationships between words regardless of their distance from one another in the text. By utilizing multi-head self-attention and positional encodings, Transformers can dynamically weigh the relevance of every token in a prompt relative to every other token, capturing deep semantic context across very long sequences. An operational caveat: although Transformers are highly parallelizable and run efficiently on GPUs, their computational complexity scales quadratically with sequence length, making them computationally intensive.

To review the distractors: the claim that the major capability is reliance on sequential, step-by-step processing of tokens to preserve word order describes recurrent layers, which process data step-by-step; this serial dependency makes training slow and limits long-range context due to vanishing gradients. The claim that the major capability is use of convolutional filters to extract local spatial features from text documents describes a convolutional approach, not the Transformer’s self-attention foundation. The claim that the major capability is a compact design requiring significantly fewer parameters and memory than traditional architectures is incorrect because Transformers are notoriously parameter-heavy, requiring massive amounts of memory and compute to train and deploy.