When pre-processing text for machine learning models, a developer decides to use dense word embeddings (such as Word2Vec or GloVe) rather than a traditional Bag-of-Words (BoW) approach. What key advantage does this choice provide?
Select an answer to reveal the explanation.
Short Explanation
You're choosing dense word embeddings because they map words into a continuous vector space where distance captures meaning and grammar. That lets your model generalize, so 'excellent' and 'outstanding' behave similarly instead of being unrelated checklist entries.
Full Explanation
In natural language processing, the Bag-of-Words model represents text by counting word frequencies. This results in high-dimensional, sparse vectors where the dimension equals the vocabulary size. One major disadvantage of Bag-of-Words is that it treats each word as an independent entity, ignoring any relationship between words. For example, in a Bag-of-Words representation, the vectors for 'happy' and 'cheerful' are orthogonal, meaning their mathematical similarity is zero.
The correct choice is the one stating that word embeddings represent words in a continuous vector space where distance captures semantic and syntactic associations. Word embeddings resolve the Bag-of-Words weakness by mapping words into a dense, lower-dimensional vector space, typically 100 to 768 dimensions. The values in these vectors are learned during training based on the contexts in which the words appear. As a result, words that share similar semantic or syntactic contexts are placed close to one another in the vector space. The geometric distance, such as cosine similarity, between vectors represents their semantic similarity. This enables machine learning models to generalize; if a model learns a pattern for 'excellent,' it can apply that same pattern to 'outstanding' because their embedding vectors reside in the same region of the vector space.
The claim that word embeddings generate highly sparse vectors that simplify the calculation of mathematical dot products is incorrect because word embeddings generate dense vectors, whereas Bag-of-Words generates sparse vectors. Sparse vectors can be harder to optimize at scale, but dense representations are what allow semantic clustering.
The claim that word embeddings guarantee that the model will process text faster because they bypass tokenization is incorrect because computing and lookup of dense embeddings is mathematically more involved than simple frequency counts, so it is not inherently faster to construct initially.
The claim that word embeddings represent words as human-readable rule sets that explicitly define dictionary concepts is incorrect because dense vector values, such as [0.25, -0.11, 0.89...], are not directly human-interpretable in the way that rule-based systems or plain word checklists are.