During the translation of a long paragraph, a Transformer-based machine translation system needs to translate the pronoun "it" in the final sentence. To resolve what "it" refers to, which mechanism does the model rely on to weigh the relevance of preceding nouns in the paragraph?
Select an answer to reveal the explanation.
Short Explanation
When you need to figure out what "it" refers to, you rely on self-attention to score every earlier token and focus on the most relevant one, no matter how far back it is. That keeps long-term dependencies intact instead of letting context fade like in older sequential models.
Full Explanation
The self-attention mechanism is the core component that enables Transformers to process natural language effectively. In sequence-to-sequence tasks, resolving context-dependent words such as pronouns, homonyms, or verbs with distant subjects requires looking at other parts of the text. Traditional architectures like RNNs process tokens sequentially, creating a bottleneck where historical context must be compressed into a single hidden vector. This often leads to loss of information over long sequences.
Self-attention resolves this by calculating query, key, and value vectors for each token in the sequence. For any given token, the model computes a dot-product between its query vector and the key vectors of all other tokens in the sequence. These scores are normalized using a softmax function to produce attention weights. The model then computes a weighted sum of the value vectors based on these weights. This process allows the model to dynamically focus on the most relevant tokens in the input sequence, preserving long-range syntactic and semantic dependencies without being limited by sequential distance.
The sequence truncation block that discards older tokens to prevent memory leaks is incorrect because truncation is a method of dropping data, not a mechanism for weighing context or handling dependencies within the retained text. The linear activation function that maps word embeddings directly to a predefined subject-verb concordance database is incorrect because neural networks do not rely on static concordance databases for translation; they learn context representations dynamically. The feed-forward network layer that compresses the sequence length into a single average token representation is incorrect because the feed-forward network operates on each token position independently and does not calculate relations across different positions in the sequence.