What role does multi-head attention play in a Transformer architecture?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Picture a single attention head as one person reading a sentence and noticing just one kind of relationship — say, subject-verb agreement. Multi-head attention is like putting several readers on the job at once, each looking through a different lens (one might track syntax, another long-range references, another local word order), and then combining what they all found. Formally, that's exactly what it does: jointly attend to information from different representation subspaces at different positions. It's not a parameter-saving trick — in fact it adds more attention computations, not fewer, by running several heads in parallel. It doesn't replace positional encoding either; Transformers still need positional encoding separately since attention alone has no built-in sense of token order. And it has nothing to do with one-hot encoding, which is just a way of representing categorical tokens before they even reach the attention mechanism.
Full explanation below image
Full Explanation
Multi-head attention, introduced in the original Transformer paper ('Attention Is All You Need'), extends the basic scaled dot-product attention mechanism by running multiple attention operations ('heads') in parallel, each with its own learned linear projections of the queries, keys, and values into different representation subspaces. This allows the model to jointly attend to information from different representation subspaces at different positions simultaneously — for example, one head might learn to capture short-range syntactic dependencies while another captures long-range semantic relationships, and the outputs of all heads are concatenated and linearly projected to form the final attention output. This design gives the Transformer significantly more representational flexibility than a single attention head could provide, since a single head's softmax-weighted average can only capture one type of relationship pattern per layer, whereas multiple heads let the model learn several complementary patterns concurrently and combine them. The first distractor, reducing total parameters by sharing a single attention weight, is incorrect and in fact describes the opposite effect — multi-head attention typically increases the number of learnable parameters relative to a single large attention head of comparable total dimensionality (since each head has its own set of query/key/value projection matrices), and there is no single shared attention weight being reused across the network. The second distractor, replacing the need for positional encoding, is incorrect because attention (including multi-head attention) is inherently permutation-invariant — it has no built-in notion of sequence order — which is exactly why Transformers must add explicit positional encodings (or use relative position representations) to inject order information; multi-head attention does not substitute for this requirement. The third distractor, converting input tokens into one-hot encoded vectors, describes a token representation step (or the precursor to an embedding lookup) that happens well before any attention computation, and is unrelated to what attention mechanisms do; attention operates on continuous embedding vectors, not on one-hot encodings, and is not responsible for creating them. Multi-head attention's ability to model diverse relationships in parallel is a key reason Transformers achieve such strong performance on sequence modeling and generative tasks compared to earlier architectures.