You are preparing a large dataset of unstructured text reviews for a sentiment classification model. To reduce noise and consolidate vocabulary, you want to strip out highly frequent, uninformative words (like 'and', 'the', 'is') and reduce words like 'running', 'runs', and 'ran' to their base dictionary form. Which text preprocessing techniques should you apply?
Select an answer to reveal the explanation.
![A diagram showing a text preprocessing pipeline. A raw text box containing 'The actors were running and gave their best performance' goes through 'Stop Word Removal' (removing 'The', 'were', 'and', 'their') and 'Lemmatization' (converting 'running' to 'run', 'gave' to 'give', 'best' to 'good'), outputting the clean list: ['actor', 'run', 'give', 'good', 'performance'].](/media/blockchain-council/caid/CAID-e6-q256-infographic.webp)
Short Explanation
Check this out: computers are great with numbers, but they're terrible at understanding human language. If you feed raw movie reviews straight into a model, it gets overwhelmed by noise. Think of all those filler words like 'the,' 'a,' and 'is'—they don't tell us anything about whether a movie was good or bad. We call those stop words, and we want them gone! Next, look at words like 'running,' 'runs,' and 'ran.' They all mean the same basic thing. By using lemmatization, we reduce them to their base form: 'run.' This shrinks our vocabulary and makes the model's job a whole lot easier. You wouldn't use PCA or K-Means here—those are for numerical data and clustering. To clean up raw text and get it ready for classification, lemmatization and stop word removal are your go-to tools!
Full Explanation
Text preprocessing is a foundational step in natural language processing (NLP) that transforms raw, unstructured text into a clean format suitable for machine learning models. When building a sentiment classifier, two of the most critical preprocessing steps are stop word removal and lemmatization. Stop words are highly frequent words in a language (such as 'the', 'is', 'at', 'which', and 'on') that carry very little semantic meaning or sentiment value. Removing them dramatically reduces the size of the vocabulary and minimizes noise, allowing the model to focus on content-carrying words (like 'excellent', 'dreadful', or 'disappointed'). Lemmatization is the process of grouping together the inflected forms of a word so they can be analyzed as a single item, identified by the word's lemma (its dictionary form). For example, the words 'better' and 'best' are mapped to 'good', and 'computes', 'computing', and 'computed' are mapped to 'compute'. Unlike stemming—which crudely chops off the ends of words—lemmatization uses vocabulary and morphological analysis to return valid base words. This consolidates words with the same underlying meaning, reducing sparsity in the data. Let's look at why the other options are incorrect: Principal Component Analysis (PCA) is a dimensionality reduction technique used on continuous numerical datasets, not raw text; One-hot encoding is used to represent categorical variables as binary vectors, but it does not consolidate words or remove noise from text; K-Means is an unsupervised clustering algorithm, not a text preprocessing step. Therefore, lemmatization and stop word removal are the key preprocessing tasks for cleaning text data.