You're preparing a dataset of text reviews for a sentiment classification model. To clean the text and reduce the overall size of the vocabulary, you need to strip out common filler words (like "the" and "is") and convert verbs and nouns back to their root dictionary form. Which preprocessing techniques should you apply?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out — when you're analyzing text, words like "the", "is", and "at" don't carry any emotional weight. They're just filler. We call these stop words, and we want them gone so our model can focus on the words that actually matter, like "excellent" or "terrible". We also want to group different forms of the same word — like "running", "runs", and "ran" — back to their base dictionary form, which is "run". That process is called lemmatization. Together, these steps clean up your text and keep your vocabulary size manageable. Got it? Sweet. Let's keep rolling.
Full explanation below image
Full Explanation
In Natural Language Processing (NLP), text preprocessing is a critical step to convert raw, unstructured text into a clean format suitable for machine learning models. For tasks like sentiment analysis, two foundational preprocessing techniques are stop-word removal and lemmatization.
Stop-word removal targets words that appear frequently in a language but carry minimal semantic value for sentiment (e.g., articles like "the", "a", prepositions like "in", "under", and auxiliary verbs like "is", "was"). Removing them reduces noise and allows the model to focus on content-rich terms that dictate sentiment (e.g., adjectives like "fantastic" or "disastrous").
Lemmatization is the process of grouping together the inflected forms of a word so they can be analyzed as a single item, identified by the word's lemma (its dictionary form). Unlike stemming, which crudely chops off word endings, lemmatization uses vocabulary and morphological analysis to accurately return roots (e.g., converting "better" to "good" or "studies" to "study"). This reduces the dimensionality of the text representation and helps the model generalize across different tenses and pluralities.
Let's look at the distractors: One-hot encoding (Option A) represents categories as binary vectors but does not clean text or reduce vocabulary size when applied to raw text directly. Principal Component Analysis (PCA) (Option B) is a mathematical dimensionality reduction technique for numeric matrices, not a text-cleaning step. K-Means clustering (Option D) is an unsupervised learning algorithm, not a text preprocessing technique. Thus, lemmatization and stop-word removal are the correct preprocessing steps for this NLP task.