When a Transformer model is generating a summary of a long document, how does the self-attention mechanism determine which parts of the original text are most critical for the current summary token?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal—when you're summarizing a five-page document, you don't read the whole thing and then write the summary from memory without looking back. As you write each sentence of your summary, your eyes are constantly scanning back to the original text, focusing on specific key terms, numbers, or names. That's exactly how self-attention works during text generation. Instead of using a fixed formula that treats every word the same, or compressing everything into a single hidden state like an old RNN, the Transformer calculates a dynamic weight distribution. This weight distribution tells the model exactly which words in the original document are important right now. If it's writing about a company's revenue, the attention weights will light up the numbers in the original text. If it's writing about a person, the attention weights will focus on the name. This dynamic weighting is what makes Transformers so incredibly flexible and accurate at handling long-term dependencies. Make sure you remember: attention is all about dynamically weighing relevance, not just applying a fixed filter or compression!
Full Explanation
In sequence-to-sequence tasks like document summarization, the model must map a long source sequence to a shorter target sequence. The self-attention mechanism achieves this by dynamically calculating the relevance of each input token relative to every other token.
Mathematically, self-attention uses three vectors derived from the input embeddings: Queries (Q), Keys (K), and Values (V). To determine the importance of input tokens, the model computes the dot product of the Queries and Keys, scales the result, and applies a softmax function to obtain a probability distribution (the attention weights). These weights represent how much "attention" the model should pay to each token in the sequence. Finally, the model computes a weighted sum of the Values based on these weights. Because this weight distribution is calculated dynamically at runtime for each step of generation, the model can adjust its focus to different parts of the source text depending on the token it is currently producing, successfully preserving long-range dependencies.
Let's review the incorrect options: - Option B is incorrect because self-attention does not downsample or discard words based on static part-of-speech rules; it processes all tokens and weighs their relevance dynamically. - Option C is incorrect because recurrent feedback loops are the defining feature of RNNs, which Transformers specifically avoid to enable parallel training. - Option D is incorrect because 2D convolutions are used in spatial computer vision applications and do not calculate dynamic attention weights across token sequences.