When a Transformer model is generating a summary of a long document, how does the self-attention mechanism determine which parts of the original text are most critical for the current summary token?
Select an answer to reveal the explanation.
Short Explanation
When you're summarizing a long document, self-attention doesn't treat every word the same or squeeze the whole text into one fixed state. It keeps looking back at the source and assigns dynamic weights to the tokens that are most relevant to the summary token you're generating right now. That dynamic relevance weighting is what lets Transformers handle long-term dependencies.
Full Explanation
In sequence-to-sequence tasks like document summarization, the model must map a long source sequence to a shorter target sequence. The self-attention mechanism achieves this by dynamically calculating the relevance of each input token relative to every other token, including the current generation state.
Mathematically, self-attention uses three vectors derived from the input embeddings: query, key, and value vectors. To determine the importance of input tokens, the model computes the dot product of the query and key vectors, scales the result, and applies a softmax function to obtain a probability distribution, which becomes the attention weights. These weights represent how much attention the model should pay to each token in the sequence. Finally, the model computes a weighted sum of the value vectors based on these weights. Because this weight distribution is calculated dynamically at runtime for each step of generation, the model can adjust its focus to different parts of the source text depending on the token it is currently producing, successfully preserving long-range dependencies. Remember: attention is all about dynamically weighing relevance, not just applying a fixed filter or compression.
The approach that applies a fixed linear projection to downsample the input sequence by removing adjectives and adverbs is incorrect because self-attention does not downsample or discard words based on static part-of-speech rules; it processes all tokens and weighs their relevance dynamically. The approach that routes tokens through a recurrent feedback loop storing context in a hidden state vector is incorrect because recurrent feedback loops are the defining feature of recurrent neural networks, which Transformers specifically avoid to enable parallel training. The approach that computes the spatial distance of characters using a two-dimensional convolution kernel is incorrect because two-dimensional convolutions are used in spatial computer vision applications and do not calculate dynamic attention weights across token sequences.