While preparing a text dataset for a machine learning model, you decide to filter out words like "the", "is", "at", and "which" because they appear frequently but carry very little semantic information. What is this preprocessing step called?
Select an answer to reveal the explanation.
Short Explanation
Think of it like this: if you're taking notes during a fast-paced technical lecture, you don't write down every single word the speaker says. You skip words like "of", "the", "at", and "is" because they don't carry any real technical weight. You only write down the nouns, verbs, and key terms. In NLP, those filler words are called "stop words". They're grammatical glue, but when we're trying to figure out what a document is about, they just get in the way and waste compute power. So, we strip them out! This process is called removing stop words. Don't confuse it with tokenization (which is just cutting text into words) or stemming (which is chopping words down to their roots). Let's keep rolling!
Full Explanation
Stop words are high-frequency words that are commonly filtered out during text preprocessing in NLP pipelines. While crucial for grammatical coherence in human communication, they lack unique semantic content for information retrieval and document classification. - Option B is correct. Removing stop words is the process of discarding these common, low-information words from the dataset to focus analysis on semantically dense terms. - Option A is incorrect. Tokenization is the initial step of splitting a string of text into smaller units (tokens), such as words or punctuation, but does not filter out any data. - Option C is incorrect. Stemming reduces words to their word stem or root form (e.g., 'arguing' to 'argu') but does not delete words from the corpus. - Option D is incorrect. Part-of-speech tagging is the process of marking up a word in a text as corresponding to a particular part of speech (noun, verb, adjective, etc.) based on its definition and context.