A product team fine-tunes a Transformer for sentence translation. Which mechanism primarily lets the model relate distant words in long inputs without relying only on a single compressed recurrent state?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's where it gets interesting. Old seq2seq often squeezed a whole sentence into one overloaded memory cell—like stuffing a novel into a sticky note. Transformers use attention: every word can look at every other word and decide what matters. "It" at the end of a long sentence can still lock onto the noun way back near the start. Boss wants better translation of long product manuals? Attention is the lever, not "just backprop" or "add more sigmoids." Exam trap: backprop trains the model but isn't the sequence mechanism. Land the takeaway: attention weighs all positions so long-range links stay reachable. Stick with me—this is core exam material.
Full explanation below image
Full Explanation
Transformer models handle long sentences primarily through attention mechanisms rather than through a single recurrent summary state. In self-attention, queries, keys, and values derived from token representations produce a weighted combination of values, so each position can gather information from any other position in the sequence, subject to masking rules. In encoder-decoder translation, cross-attention further lets decoder states attend over encoder outputs at every generation step. Because connections are direct rather than routed only through a chain of recurrent updates, gradients and contextual information need not travel exclusively through one fixed-size hidden vector, which historically limited vanilla encoder-decoder recurrent networks on long inputs and caused lost alignments.
Backpropagation is essential for learning attention weights and feedforward parameters, but it is a training algorithm shared by convolutional nets, recurrent nets, and Transformers alike. It is therefore not the architectural mechanism that models long-range token relationships. Sigmoid activations squash individual unit outputs into a unit interval; they do not create content-based pairwise routing across sequence positions. Relying on a small fixed context vector without attention reintroduces the very bottleneck that attention was designed to relieve in neural machine translation research.
In practice, multi-head attention lets different heads specialize in phenomena such as syntax, coreference, or translational alignment, while positional encodings or relative position biases inject order because pure attention is permutation-equivariant without them. Computational cost grows with sequence length, so efficient attention variants and length limits appear in production systems, yet the exam-level principle remains stable: weigh all relevant tokens by learned importance. A durable memory aid is that recurrent networks walk step by step while attention can connect distant tokens in one hop. When asked what lets Transformers manage long translation inputs in terms of dependency modeling, choose attention—not generic training algorithms, elementwise activations, or compressed fixed states alone.