A data scientist is preprocessing a text corpus using the Natural Language Toolkit (NLTK) in Python. They need to reduce inflected word forms (like "running", "runs", and "ran") to their base dictionary form. Which NLTK module should they import?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of it like this: if you're searching a database for the word "organize", you also want to find documents that contain "organizes", "organized", or "organizing". To do this in NLP, we need to strip away the prefixes and suffixes to get back to the core root word. In the Natural Language Toolkit (NLTK), the module that does this heavy lifting is nltk.stem. It contains stemmers (like PorterStemmer) and lemmatizers (like WordNetLemmatizer) that do exactly this job. Don't confuse it with nltk.tokenize, which just splits your sentences into individual words, or nltk.tag, which labels parts of speech. Pay close attention here, because getting your preprocessing right is half the battle in NLP. Got it? Sweet. Let's keep rolling.
Full explanation below image
Full Explanation
In NLP, inflected forms of words are normalized to a common base form to reduce vocabulary dimensionality and improve matching. This is done via stemming (removing endings heuristically) or lemmatization (using morphological analysis and vocabulary to find the dictionary form, or lemma). - Option D is correct. The nltk.stem module in NLTK contains classes and functions for both stemming (e.g., PorterStemmer, SnowballStemmer) and lemmatization (e.g., WordNetLemmatizer). - Option A is incorrect. The nltk.tokenize module is used to split continuous strings of text into individual units or tokens (words, sentences, or characters), not to reduce them to their base form. - Option B is incorrect. The nltk.tag module is used for Part-of-Speech (POS) tagging, which labels words with their grammatical category (noun, verb, etc.) based on context. - Option C is incorrect. The nltk.corpus module provides interfaces to read-only corpora and lexical resources (like the Brown Corpus or WordNet database files) but does not contain the stemmer or lemmatizer processing classes.