When constructing a deep learning model for text analysis in PyTorch, a developer includes the torch.nn.Embedding module as the first layer. What is the primary function of this module during the forward pass?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Let's get practical with PyTorch. If you're building a neural network to process text, you can't just feed raw strings or even a list of word ID numbers directly into your linear layers. The math just doesn't work. You need to convert those integer word IDs into dense vectors of floating-point numbers. That's exactly what torch.nn.Embedding does. Think of it like a giant lookup dictionary. When a sequence of word IDs (like [12, 452, 9]) comes in, the embedding layer goes to its internal table and retrieves a pre-configured vector of, say, 300 numbers for each ID. These vectors aren't sparse—they aren't full of zeros like a Bag-of-Words representation. They are dense, fixed-size vectors where the numbers are constantly updated during backpropagation as the model learns. So, it's basically a lookup table that maps discrete word tokens into continuous space. Keep this straight for the exam: torch.nn.Embedding is all about taking integer IDs and turning them into dense, fixed-size vectors.
Full explanation below image
![A diagram showing an input tensor of integer indices (e.g., [4, 1]) indexing into a 2D weight matrix (Embedding Lookup Table). The output shows two dense vectors of length N (e.g., [0.12, -0.45, ...] and [0.78, 0.03, ...]) being extracted, demonstrating how indices map to dense continuous rows.](/media/blockchain-council/caid/CAID-q167-infographic.webp)
Full Explanation
In PyTorch, the torch.nn.Embedding module acts as a lookup table that maps discrete tokens (represented by integer indices) to dense, continuous vectors. When building models for NLP, input text is first tokenized, and each unique token is assigned an integer index from a vocabulary dictionary. However, integer values carry no inherent semantic meaning (e.g., index 5 is not mathematically related to index 6).
The embedding layer translates these sparse integer indices into dense, floating-point vectors of a fixed size (the embedding dimension, e.g., 128, 256, or 512). The module initializes a weight matrix of shape (num_embeddings, embedding_dim). During the forward pass, the input tensor of token indices is used to index into this weight matrix, retrieving the corresponding row vector for each token. Importantly, these embedding vectors are learnable parameters. During model training, backpropagation adjusts these vector values, allowing the network to group semantically similar words closer together in the vector space.
Let's examine the incorrect options: - Option A is incorrect because text classification is typically performed by the final output layer (e.g., a linear layer followed by softmax), not the initial embedding layer. - Option C is incorrect because spatial filtering across sequences is the job of convolutional layers (torch.nn.Conv1d), not embedding layers. - Option D is incorrect because calculating attention weights is the role of attention layers (torch.nn.MultiheadAttention), not standard embeddings.