What does the self-attention mechanism in a Transformer architecture enable the model to do?
Select an answer to reveal the explanation.
Short Explanation
Check this out: self-attention lets you weigh every word against every other word at once, so the model can capture long-range context instead of crawling through text one token at a time. It’s how the model figures out relationships like whether 'bank' means a river bank or a money bank.
Full Explanation
The self-attention mechanism is the defining innovation of the Transformer architecture, which replaced Recurrent Neural Networks (RNNs) as the state-of-the-art framework for sequential tasks. Traditional RNNs process sequences token-by-token, which creates a computational bottleneck and makes it difficult to capture relationships between distant words, a problem often associated with the vanishing gradient problem. In a long sentence, the model could lose track of what happened at the beginning by the time it got to the end.
Self-attention solves this by processing all tokens in a sequence simultaneously in a non-sequential, parallel fashion. For each token, the mechanism calculates a query, key, and value vector. It computes similarity scores between the token's query and the keys of all other tokens in the sequence, producing a set of attention weights. These weights determine how much focus the model should place on other parts of the sequence when encoding the current token, allowing it to capture long-range contextual relationships and dependencies instantly. For example, when the model looks at the word 'bank,' it can compare it with all other words at the same time to help decide whether it refers to a river bank or a money bank.
The description of processing tokens sequentially, step-by-step, to maintain a chronological memory of the text is incorrect because sequential, token-by-token processing is characteristic of RNNs and LSTMs. Transformers avoid sequential processing to enable parallelization on GPUs during training.
The description of removing common stop words and punctuation marks automatically to simplify the input sequence is incorrect because filtering out stop words and punctuation is a basic text preprocessing step that occurs before the text is embedded and fed into the model. Transformers typically process the complete text, including punctuation, to preserve linguistic structure.
The description of compressing high-dimensional input vectors into a lower-dimensional latent space to speed up processing is incorrect because compressing high-dimensional vectors to a lower-dimensional latent space is the function of dimensionality reduction layers, such as linear projections, bottleneck layers, or autoencoders, not the self-attention mechanism.