While cleaning a text dataset, an analyst notices many low-information function words such as "the," "a," and "is." Which preprocessing action specifically targets those terms?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Those little words—"the," "a," "is"—show up everywhere and often don't help a simple classifier decide topics or spam. Stopping them at the door is called stop-word removal. Tokenization just chops the sentence into pieces; lemmatization cleans "running" to "run"; neither is "delete the boring glue words." Exam trap: people mix all preprocessing terms into one soup. Separate the jobs in your head. If the stem points at useless frequent function words, the move is stop-word removal. Short, sweet, and very testable.
Full explanation below image
Full Explanation
Stop words are high-frequency function words—articles, auxiliary verbs, common prepositions—that often contribute limited discriminative value in traditional bag-of-words or TF-IDF models. Stop-word removal is the preprocessing step that filters those tokens using a language-specific list or frequency heuristics. In the scenario where the analyst flags words like "the," "a," and "is" as adding little meaning, stop-word removal is the targeted action. The motivation is practical: those tokens inflate vocabulary size and document length without usually helping a linear or tree-based text classifier separate classes, so removing them can shrink features and speed training while focusing weight on content-bearing terms.
Tokenization is a prerequisite that segments raw character streams into tokens, but segmentation alone retains stop words until a separate filter runs. You still need tokenization first; the point is that tokenization and stop-word removal solve different problems. Lemmatization maps inflected forms to canonical lemmas (for example, "better" to "good" in careful systems, or "cars" to "car"), reducing morphological variation rather than deleting function words. One-hot encoding converts categorical variables into binary indicator vectors; it is unrelated to pruning English function words from free text and belongs to a different stage of feature engineering when labels or discrete attributes must be numeric.
Caveats matter in practice: for sentiment, sarcasm, negation scope, or transformer-based models, stop words sometimes carry signal, and aggressive lists can harm performance. Domain-specific stoplists may differ from generic English lists, and some pipelines replace hard deletion with down-weighting via TF-IDF. Still, certification items that describe meaningless frequent glue words are pointing squarely at stop-word removal. Pair the vocabulary carefully: tokenize means split into units; lemmatize or stem means normalize surface forms; remove stop words means drop low-content frequent tokens. That checklist prevents mix-ups under exam pressure and keeps feature spaces tighter for classical NLP baselines. When in doubt on a stem about "the/a/is," choose stop-word removal rather than neighboring preprocessing terms.