During the translation of a long paragraph, a Transformer-based machine translation system needs to translate the pronoun "it" in the final sentence. To resolve what "it" refers to, which mechanism does the model rely on to weigh the relevance of preceding nouns in the paragraph?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out—imagine you're reading a book and you come across the word "it." To know what "it" means, your brain automatically jumps back a couple of sentences to find the subject. If the book talked about a dog three pages ago, and now says "it barked," you immediately make the connection. That's exactly what the attention mechanism does for neural networks! In old RNNs, by the time the model got to the end of a long paragraph, the memory of that "dog" at the beginning had faded away completely. But in a Transformer, the self-attention mechanism calculates a score between every single word in the sequence. When the model processes the word "it," the attention mechanism says, "Hey, look over here, there's an 85% match with the word 'dog' from earlier." This allows the model to keep those crucial long-term dependencies intact, no matter how long the text gets. It's not deleting words or doing simple database lookups; it's dynamically weighing what's important at any given moment. Remember this for the exam: attention is all about focus and capturing those far-apart relationships.
Full explanation below image
Full Explanation
The attention mechanism, specifically self-attention, is the core component that enables Transformers to process natural language effectively. In sequence-to-sequence tasks, resolving context-dependent words (such as pronouns, homonyms, or verbs with distant subjects) requires looking at other parts of the text. Traditional architectures like RNNs process tokens sequentially, creating a bottleneck where historical context must be compressed into a single hidden vector. This often leads to loss of information over long sequences.
Self-attention resolves this by calculating query, key, and value vectors for each token in the sequence. For any given token, the model computes a dot-product between its query vector and the key vectors of all other tokens in the sequence. These scores are normalized using a softmax function to produce attention weights. The model then computes a weighted sum of the value vectors based on these weights. This process allows the model to dynamically focus on the most relevant tokens in the input sequence, preserving long-range syntactic and semantic dependencies without being limited by sequential distance.
Let's look at the incorrect options: - Option A is incorrect because sequence truncation is a method of dropping data, not a mechanism for weighing context or handling dependencies within the retained text. - Option C is incorrect because neural networks do not rely on static concordance databases for translation; they learn context representations dynamically. - Option D is incorrect because the feed-forward network operates on each token position independently and does not calculate relations across different positions in the sequence.