Computers are great at math, but they don't understand human words out of the box. In modern Natural Language Processing (NLP), how do we represent words so that models can process them mathematically while capturing their semantic meanings and relationships?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal—computers don't know what 'king' or 'queen' means, so you turn words into dense numeric vectors that put similar meanings close together. That's word embeddings, and it lets models do math with language instead of just matching strings.
Full Explanation
In Natural Language Processing (NLP), word embedding is a representation learning technique where words from a vocabulary are mapped to vectors of real numbers in a continuous vector space. Historically, simple representation techniques like one-hot encoding were used. In a one-hot representation, each word is represented as a sparse vector with a dimension equal to the size of the vocabulary; with a vocabulary of 50,000 words, that means a vector with 49,999 zeros and a single one. This approach has two major flaws: it suffers from extreme high-dimensionality and sparsity (the curse of dimensionality), and it fails to capture semantic similarity, as the dot product between any two distinct one-hot vectors is always zero, meaning the computer treats 'cat' and 'kitten' as completely unrelated. Word embeddings (such as Word2Vec, GloVe, or FastText) resolve these issues by representing words in a dense, lower-dimensional space (typically 100 to 300 dimensions). Because the vectors are learned based on the context in which words appear (the distributional hypothesis: words that occur in similar contexts tend to have similar meanings), semantically related words are positioned near each other in the vector space. This spatial arrangement allows the model to perform vector arithmetic that reflects semantic relationships, such as the classic example: Vector('King') - Vector('Man') + Vector('Woman') ≈ Vector('Queen'). The text-to-speech engine approach converts written text into spoken audio, which is an output generation task rather than an NLP input representation method. The regular-expression cleanup approach, stripping punctuation and stop words, is a basic text preprocessing step, not a mathematical representation technique. The one-hot encoding approach represents words as numbers, but it creates sparse, high-dimensional, independent binary vectors that fail to capture semantic relationships. Therefore, mapping words to dense, low-dimensional vectors of real numbers in a continuous space is the main purpose of word embeddings.