What does the self-attention mechanism in a Transformer architecture enable the model to do?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: in the old days, recurrent neural networks (RNNs) had to process text one word at a time. It was like reading a book through a tiny straw. If you had a long sentence, the model would completely forget what happened at the beginning by the time it got to the end. Transformers fixed this with self-attention. Think of it like a party where everyone is talking, and the model decides who to pay attention to for every single word. When the model looks at the word 'bank,' it checks all the other words in the sentence to figure out if it means a river bank or a money bank. It does this for all words at the same time, capturing long-range connections easily. That's why C is the winner. Option A describes RNNs, Option B is basic preprocessing, and Option D is dimensionality reduction. Self-attention is the secret sauce of modern AI!
Full explanation below image
Full Explanation
The self-attention mechanism is the defining innovation of the Transformer architecture, which replaced Recurrent Neural Networks (RNNs) as the state-of-the-art framework for sequential tasks. Traditional RNNs process sequences token-by-token, which creates a computational bottleneck and makes it difficult to capture relationships between distant words (the vanishing gradient problem).
Self-attention solves this by processing all tokens in a sequence simultaneously in a non-sequential, parallel fashion. For each token, the mechanism calculates a query, key, and value vector. It computes similarity scores between the token's query and the keys of all other tokens in the sequence, producing a set of attention weights. These weights determine how much focus the model should place on other parts of the sequence when encoding the current token, allowing it to capture long-range contextual relationships and dependencies instantly.
Let's analyze why the other options are incorrect: - Option A is incorrect because sequential, token-by-token processing is characteristic of RNNs and LSTMs. Transformers avoid sequential processing to enable parallelization on GPUs during training. - Option B is incorrect because filtering out stop words and punctuation is a basic text preprocessing step that occurs before the text is embedded and fed into the model. Transformers typically process the complete text, including punctuation, to preserve linguistic structure. - Option D is incorrect because compressing high-dimensional vectors to a lower-dimensional latent space is the function of dimensionality reduction layers (like linear projections, bottleneck layers, or autoencoders), not the self-attention mechanism.