When preprocessing text for a machine learning model, a developer decides to represent words using one-hot encoding. Which of the following best describes this representation and its primary drawback?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of one-hot encoding like a huge light switchboard where only one switch can be turned on at a time. If you have a vocabulary of 10,000 words, every single word is represented by a vector that is 10,000 items long! For the word 'router', the switch at position 5 might be '1' and all other 9,999 positions are '0'. That's a lot of wasted space! We call this a sparse vector. The big downside here is high dimensionality—your vectors get huge, and your machine's memory gets eaten up fast. Option D captures this perfectly.
Full explanation below image
![A matrix diagram showing a vocabulary of five words: ['apple', 'banana', 'cherry', 'date', 'elderberry']. Below it, show the one-hot vectors for 'apple' as [1, 0, 0, 0, 0] and 'cherry' as [0, 0, 1, 0, 0], highlighting the single '1' in each sparse vector and the total dimension of 5.](/media/blockchain-council/caid/CAID-q155-infographic.webp)
Full Explanation
One-hot encoding is a classic technique used to convert categorical variables, such as words in a vocabulary, into a numerical format that machine learning algorithms can process. In this scheme, a vocabulary of size $V$ is established. Each unique word is assigned a vector of length $V$. In this vector, the index corresponding to the word's position in the vocabulary is set to 1, while all other $V-1$ elements are set to 0.
While simple to implement, one-hot encoding has two major limitations. First, it leads to high-dimensional, sparse representations. If a corpus contains 50,000 unique words, each word is represented by a 50,000-dimensional vector consisting almost entirely of zeros. This is computationally expensive and memory-inefficient. Second, one-hot encoding treats all words as equidistant and independent, completely failing to capture semantic relationships (e.g., the vector for 'cat' has no more similarity to 'dog' than it does to 'refrigerator').
Let's analyze the incorrect options: - Option A describes word embeddings (like Word2Vec, GloVe, or FastText), which represent words as low-dimensional dense vectors of continuous values, not sparse one-hot vectors. - Option B describes stop word removal, which is a preprocessing step to remove frequent, low-information words, not a vector encoding scheme. - Option C describes label encoding or integer encoding, where each word is mapped directly to a single integer (e.g., 'cat' = 1, 'dog' = 2), which is not a vector representation.
Remember for the exam: one-hot encoding creates a sparse vector of size $V$ where a single element is 1, leading to sparse representation and high dimensionality.