In an NLP model, raw text is first tokenized into integer IDs and then passed through an embedding layer before reaching the rest of the network. What role does that embedding layer serve?
Select an answer to reveal the explanation.
Short Explanation and Infographic
An embedding layer is basically a lookup table that turns each token ID — just an arbitrary integer, no inherent meaning — into a dense vector of real numbers, and crucially those vectors are learned so that words used in similar contexts end up close together in that vector space. That's what makes text usable by a neural network at all, since the network can't do useful math on a raw index like '482' representing the word 'dog.' It's not doing text cleanup like stripping stop words or punctuation — that happens earlier, in preprocessing. It's not applying softmax either; softmax shows up at the output layer for producing a probability distribution over next-word predictions, which is a completely different stage of the pipeline. And it's not about merging rare words into an 'unknown' token — that's a vocabulary-size decision made during tokenization, before the embedding layer ever sees the data.
Full explanation below image
![An infographic showing the pipeline: raw text 'the dog runs' -> tokenizer producing integer IDs [12, 482, 91] -> arrow into an embedding lookup table -> each ID mapped to a dense vector of numbers, then a 2D scatter plot showing similar words like 'dog' and 'cat' clustered close together in vector space.](/media/deep-learning/cdls/CDLS-q045-embedding-layer-purpose-infographic.webp)
Full Explanation
The correct answer is that an embedding layer converts discrete token IDs into dense, continuous-valued vectors that the rest of the network can meaningfully operate on. After tokenization, each word or subword is represented as an integer index into a vocabulary, but that index is arbitrary — the numeric distance between token 5 and token 500 carries no semantic meaning, and neural networks need inputs where numeric relationships correspond to something useful. The embedding layer solves this by maintaining a learnable weight matrix where each row is a dense vector for one vocabulary entry; during training, these vectors are adjusted so that tokens appearing in similar contexts end up with similar vector representations, which is why classic embeddings like word2vec or GloVe famously place 'king' and 'queen' near each other and support vector arithmetic like king minus man plus woman approximating queen. This dense representation is far more compact and information-rich than a one-hot encoding and gives downstream layers (RNNs, transformers, or feedforward layers) a numerically meaningful starting point. The first distractor, removing stop words and punctuation, describes a text-preprocessing or cleaning step that happens before tokenization, not something the embedding layer itself does — by the time tokens reach the embedding layer, the vocabulary and token sequence have already been finalized, so the embedding layer has no role in filtering content. The second distractor, applying softmax to predict the next word, describes the function of an output layer in a language-modeling head, which is architecturally on the opposite end of the network from the embedding layer; softmax converts a vector of raw scores into a probability distribution over possible next tokens, a task entirely separate from converting input token IDs into dense vectors. The third distractor, compressing vocabulary size by merging rare words into an unknown token, describes a vocabulary-construction or tokenization decision, typically handled by the tokenizer (for example, replacing rare words with an <UNK> token or using subword tokenization to control vocabulary size) — this happens upstream of the embedding layer and is unrelated to what the embedding layer computes once the vocabulary is fixed. A helpful mental model: think of the embedding layer as a big lookup table where every row is a coordinate in 'meaning space,' and training nudges those coordinates so that semantically related words end up as neighbors.