What is the core purpose of the self-attention mechanism inside a Transformer block?
Select an answer to reveal the explanation.
Short Explanation
Self-attention's whole job is letting each word 'look around' at every other word in the sequence and decide how much attention to pay to each one while it's being processed. Take the sentence 'The trophy didn't fit in the suitcase because it was too big' — self-attention is what lets the model figure out whether 'it' refers to the trophy or the suitcase, by weighing the relevance of every other token. It's not a dimensionality-reduction trick, it's not a convolution (no fixed sliding filter here), and it's not the masking trick used in decoder-only training to hide future tokens — that's a related but separate mechanism (causal masking) layered on top of attention in autoregressive models.
Full Explanation
Self-attention computes, for each element in a sequence, a weighted combination of all other elements' representations, where the weights reflect how relevant each other element is to the one currently being processed. Mechanically, this is done via query, key, and value vectors derived from the input embeddings: for each token, its query vector is compared (via dot product) against every token's key vector to produce attention scores, which are normalized (softmax) into weights and used to compute a weighted sum of value vectors. This lets the model dynamically capture relationships between any two positions in a sequence, regardless of their distance, which is exactly why Transformers excel at long-range dependency modeling in text.
The first distractor confuses self-attention with a dimensionality-reduction operation; while attention does involve projecting into query/key/value spaces (sometimes of different dimensionality), its purpose is relational weighting, not compression, and dimensionality reduction is not the defining goal. The second distractor describes convolution, which applies a fixed, learned filter across local, fixed-size windows of the input — the opposite of attention's dynamic, sequence-length-spanning, content-dependent weighting; Transformers were specifically designed to overcome the local-receptive-field limitation of CNNs and RNNs for sequence tasks. The third distractor describes causal (masked) self-attention, a specific variant used in autoregressive decoders (like GPT) to prevent a position from attending to future tokens during training — this masking is an added constraint on top of the attention mechanism, not attention's defining purpose, and bidirectional Transformers (like BERT's encoder) use self-attention without this masking at all.
Memory aid: 'query asks, key answers, value delivers' — self-attention is fundamentally a content-based lookup where every token asks 'who's relevant to me?' and gets back a weighted blend of everyone's information.