What core mechanism gives the Transformer architecture its ability to model relationships across an entire input sequence?
Select an answer to reveal the explanation.
Short Explanation
The Transformer's big trick is attention. Instead of marching through a sequence one token at a time like an RNN, it looks at the whole sequence at once and figures out, for every token, how much every other token matters to it. That weighting is what lets a Transformer capture long-range relationships, like connecting a pronoun way at the end of a sentence back to the noun it refers to way at the beginning, all in parallel. That's answer D. It's specifically NOT recurrence — ditching the step-by-step hidden state is exactly what makes Transformers fast and parallelizable, unlike RNNs. It's not pooling either, which just compresses information down, losing that fine-grained relationship weighting. And it's not convolution, which only looks at a local neighborhood at a time. Attention is the mechanism that made Transformers the backbone of modern NLP.
Full Explanation
The Transformer architecture is built around the attention mechanism, most notably self-attention, which allows the model to compute, for every element in a sequence, a weighted representation based on its relationship to every other element in that same sequence. Concretely, each token is projected into query, key, and value vectors, and attention scores are computed by comparing queries against keys to determine how much focus each token should place on every other token when constructing its updated representation. This allows the Transformer to directly model long-range dependencies (e.g., a word at the start of a sentence influencing the interpretation of a word many tokens later) without the information having to pass sequentially through many intermediate steps, and because attention operates on the whole sequence simultaneously, it is highly parallelizable, which is one of the key advantages that made Transformers practical to train at scale.
The first distractor describes the mechanism used by RNNs, which process sequences step by step, carrying a hidden state forward from one time step to the next; this is precisely the sequential bottleneck that Transformers were designed to avoid, since it limits parallelism and struggles with very long-range dependencies due to vanishing gradients. The second distractor describes pooling, an operation that reduces a set of values to a single summary statistic (such as a maximum), which discards the fine-grained pairwise relationships between sequence elements that attention is specifically designed to preserve and exploit. The third distractor describes convolution, which applies a fixed-size local filter to a sequence or image; while effective at detecting local patterns, a single convolutional layer has a limited receptive field and cannot directly relate distant elements without stacking many layers, unlike self-attention which relates any two positions in a single step.
A useful memory aid: RNNs process sequentially and remember through a hidden state, CNNs process locally through sliding filters, and Transformers process globally and in parallel through attention, directly weighing every pair of elements against each other regardless of their distance in the sequence.