The Transformer model revolutionized NLP by replacing older sequential architectures. Which core building block is found in both the encoder and decoder layers of a standard Transformer architecture?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Okay, let's dive into the anatomy of a Transformer. If you open up a standard Transformer block—whether it's on the encoder side or the decoder side—you're going to see a few key pieces repeated over and over. You won't find any recurrent layers (like LSTMs) because we already talked about how recurrence slows down training. And you won't find convolutional layers here, because those are for image processing. Instead, the beating heart of every Transformer block is the Multi-Head Self-Attention module. Think of "multi-head" like having a team of analysts looking at the same sentence, but each analyst is looking for something different. One analyst might focus on matching pronouns to nouns, while another focuses on verb tenses. By running these attention heads in parallel, the model gets a rich, multidimensional view of the text. Trust me, if there's one term you need to associate with Transformers on the exam, it's attention. Got it? Sweet. Let's keep rolling.
Full explanation below image
Full Explanation
The Transformer architecture, introduced in the seminal paper "Attention Is All You Need," relies on a modular block structure that repeats across both the encoder and decoder stacks. The defining characteristic of the Transformer is its complete omission of recurrence (RNNs) and convolution (CNNs) for sequence processing.
The core component within each Transformer layer is the Multi-Head Self-Attention module. This module allows the model to process input sequences in parallel by dividing the attention mechanism into multiple "heads." Each head projects queries, keys, and values into different representation spaces, allowing the model to attend to information from different representation subspaces at different positions simultaneously. For example, one head might track syntactic relations (subject-verb agreement), while another focuses on semantic associations (connecting pronouns to the correct noun phrases).
Let's review the incorrect options: - Option A is incorrect because Recurrent units (such as GRUs or LSTMs) are deliberately excluded from the Transformer architecture to allow parallel processing. - Option B is incorrect because pooling layers are primarily used in Convolutional Neural Networks (CNNs) for computer vision tasks to reduce spatial dimensions, not in standard Transformers. - Option D is incorrect because standard Transformers do not use dilated convolutional filters, which are characteristic of CNN architectures like WaveNet.