A fixed income analyst is evaluating an AI system that a vendor claims uses a 'transformer-based architecture with multi-head attention' to analyze central bank communications and predict yield curve movements. To assess the credibility of this claim, the analyst asks the vendor's engineer to explain what the attention mechanism actually enables the model to do. Which explanation is technically accurate?

The transformer architecture, introduced in the 2017 Vaswani et al. paper 'Attention Is All You Need,' replaced recurrent neural networks (RNNs) as the dominant sequence modeling architecture by solving a fundamental problem with RNNs: the vanishing gradient problem that made it difficult for RNNs to capture dependencies between tokens that were far apart in a sequence.

The attention mechanism works as follows: for each token in the input, the model computes a Query vector, a Key vector, and a Value vector via learned linear transformations. Attention scores are computed as the scaled dot product of the Query for the current token against the Keys for all other tokens. These scores are passed through a softmax function to produce a probability distribution, which is then used to compute a weighted sum of the Value vectors. The result is a context-aware representation of each token that encodes its relationship to every other token in the sequence — dynamically, based on the current input, not statically based on position.

'Multi-head' attention runs this process multiple times in parallel with different learned projections, allowing the model to simultaneously capture different types of relationships (e.g., one head might learn syntactic dependencies while another learns semantic co-references). In financial NLP applications, this means a model analyzing an FOMC statement can simultaneously track how 'inflation' is used relative to 'transitory' across the document and how 'rate path' is discussed relative to 'balance sheet' — two entirely different relationship types captured by different attention heads.

Option A is wrong because attention weights are not static and are not based on position — they are computed dynamically for each input. Position encoding is a separate mechanism added to the input embeddings to convey order information. Option C confuses attention with tokenization or compression techniques; attention does not reduce sequence length but rather allows every position to attend to every other position. Option D confuses attention with term frequency (TF-IDF) weighting from classical NLP; attention weights are context-dependent and learned during training, not derived from corpus frequency statistics.