When training a Transformer-based model like BERT, inputs are padded to ensure all sequences within a training batch have the same length. What is the function of the "attention mask" during the forward pass?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: when you train a Transformer model in batches, every sequence in that batch must be the exact same length. Since human sentences are naturally different lengths, we have to pad the shorter ones with meaningless filler tokens (usually called [PAD] or represented as 0). The problem is, if you let the model calculate self-attention on those pad tokens, it's going to waste computation and mess up the contextual meaning of the real words. That's where the attention mask comes in. Think of it like a set of blinders. It's a simple binary mask that tells the model: "Hey, focus on these real words, but ignore those padding tokens over there." Option B is the one you want.
Full explanation below image
![A matrix grid representation of self-attention showing a sentence with padding: ["The", "cat", "sat", "[PAD]"]. Highlight the attention mask column for "[PAD]" containing negative infinity values, showing that after the Softmax layer, the attention weights for those cells become zero.](/media/blockchain-council/caid/CAID-q188-infographic.webp)
Full Explanation
In Transformer architectures, batch processing requires inputs to be structured as uniform tensors of identical shapes. Because sentences vary in length, shorter sequences are padded with placeholder tokens (e.g., [PAD]). To prevent these padding tokens from influencing the representations of actual words, the model uses an attention mask. During the self-attention calculation: $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V$$ where $M$ is the attention mask matrix. For real tokens, the mask value is $0$, leaving the attention score unchanged. For padding tokens, the mask value is set to a large negative value (typically $-\infty$ or $-10^9$). When the softmax function is applied, these large negative values are mapped to an attention probability of exactly $0$. This mathematical operation ensures that the self-attention mechanism completely ignores the padding tokens, preventing them from corrupting the contextual representation of the sequence.
Let's look at the incorrect options: - Option A is incorrect because stop word removal is a text cleaning step that occurs before tokenization, not during the forward pass of the model. - Option C describes the "masked language modeling" (MLM) pre-training objective (where words are replaced with [MASK]), which is distinct from the attention mask used to hide padding. - Option D is incorrect because padding tokens cannot be physically removed from a tensor during batch processing, as tensor shapes must remain static.