When preprocessing text for a Natural Language Processing application, what is the primary objective of performing lemmatization on the corpus?
Select an answer to reveal the explanation.
Short Explanation
Check this out. You need 'ran', 'running', and 'runs' to become 'run' so your model learns one action instead of three. Lemmatization uses context and a dictionary to return real base forms, not chopped-up roots.
Full Explanation
Lemmatization is a text normalization technique in Natural Language Processing (NLP) that reduces words to their lemma, which is their canonical, dictionary, or base form. The correct objective is to resolve words to their dictionary base form, known as a lemma, based on their intended meaning and context. Unlike stemming, which relies on simple heuristic rules to chop off the ends of words, lemmatization uses vocabulary databases and morphological analysis to determine the correct base form.
For example, when lemmatizing the verb 'saw' in the sentence 'He saw a movie', the lemmatizer uses part-of-speech tagging to recognize it as a verb and resolves it to 'see'. If the sentence was 'He bought a saw', it would identify it as a noun and leave it as 'saw'. This contextual awareness ensures that the resulting lemma is always a valid word in the target language. This is crucial for search engines, topic modeling, and text classification where preserving grammatical accuracy and semantic meaning is important.
The description that analyzes and records how frequently individual words appear throughout the corpus describes term frequency counting, which is a statistical step in feature extraction, such as bag-of-words or term frequency-inverse document frequency, not a word reduction technique. The description that strips word endings using heuristic rules, often resulting in non-dictionary word roots, describes stemming, such as using a Porter Stemmer, which often trims words to non-real root forms, such as 'studying' becoming 'studi'. The description that eliminates common, low-information words like articles and prepositions from the text describes stop-word removal, which filters out high-frequency words that carry little semantic weight, such as 'is', 'the', and 'at'.
Remember for the exam: lemmatization uses morphological and contextual rules to reduce words to their actual dictionary base form, known as a lemma.