You are building an email classifier to separate spam from legitimate communications. To keep your vocabulary size manageable and prevent common, uninformative words from skewing the model's predictions, which combination of preprocessing steps should you perform?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: when you're building a spam filter, raw text is full of noise. You've got words like 'the', 'is', and 'at' showing up everywhere. These are called stop words, and they do absolutely nothing to help your model figure out if an email is trying to sell you cheap Rolexes. Plus, words like 'running', 'runs', and 'ran' all mean basically the same thing in this context. If you keep them all separate, your vocabulary explodes and your model gets bogged down. By stripping out the stop words and using lemmatization to reduce words to their base form (like turning 'running' to 'run'), you keep your data clean and your model fast. Trust me, your server's memory will thank you!
Full explanation below image
Full Explanation
Text preprocessing is a critical step in Natural Language Processing (NLP), especially for classification tasks like spam filtering. The primary goal of preprocessing is to reduce the dimensionality of the feature space while retaining the semantic core of the message. Two primary methods for achieving this are lemmatization and the removal of stop words.
Stop words are high-frequency words (such as 'and', 'the', 'a', 'in') that appear in almost all documents but carry little to no task-specific semantic weight. Removing them drastically reduces the total number of features (vocabulary size) the model must track, preventing overfitting and reducing computational resource usage.
Lemmatization is the process of grouping together the inflected forms of a word so they can be analyzed as a single item, identified by the word's lemma (its dictionary form). For example, 'organizes', 'organized', and 'organizing' are all reduced to 'organize'. Unlike stemming, which crudely chops off word endings and can result in non-words, lemmatization uses vocabulary and morphological analysis to return valid base words.
Let's review the distractors: - Option A is incorrect because one-hot encoding on its own does not reduce vocabulary size—in fact, it creates a sparse matrix that expands linearly with vocabulary size. Padding is used to standardize sequence lengths, not to shrink vocabulary. - Option C is incorrect because you cannot create a Bag-of-Words representation without first tokenizing the text into individual words or symbols. - Option D is incorrect because building a TF-IDF matrix directly from raw, unprocessed text preserves all stop words and morphological variations, leading to a massive, noisy, and inefficient feature space.