In a standard Natural Language Processing (NLP) pipeline, what preprocessing step is typically performed immediately after text has been split into tokens?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: before you can do any cool analysis or model training, you have to clean up your text. The very first thing you do is chop the raw text into individual words or subwords—that's tokenization. But once you have those tokens, they're still messy. You've got plurals, different verb tenses, and capitalization. So, the logical next step is to normalize them. That means using stemming or lemmatization to strip them down to their core base forms. You do this before you train anything or make predictions. Get your tokens clean first, then feed them to your model. Got it? Sweet.
Full explanation below image
![A linear timeline diagram of an NLP Preprocessing pipeline. The steps are shown left-to-right as: 1) Raw Text Input ('She ran fast'), 2) Tokenization (splitting into ['She', 'ran', 'fast']), 3) Token Normalization (Lemmatization/Stemming converting ['She', 'run', 'fast']), and 4) Vectorization (converting to numbers). Step 3 is highlighted with an arrow indicating it immediately follows Step 2.](/media/blockchain-council/caid/CAID-q177-infographic.webp)
Full Explanation
In an NLP pipeline, tokenization is the foundational step where raw character sequences are parsed and split into individual structural units called tokens (typically words, subwords, or characters). Once tokenization is complete, the tokens must undergo normalization before they are converted into numerical representations for model training or inference.
This normalization step typically involves stemming or lemmatization. - Stemming is a heuristic process that chops off the ends of words (e.g., converting 'flying', 'flew', and 'flies' to 'fli'). - Lemmatization uses vocabulary databases and morphological analysis to return the base dictionary form, or lemma, of a word (e.g., converting 'better' to 'good' or 'running' to 'run'). Performing this immediately after tokenization ensures that subsequent vectorization steps (such as Bag-of-Words, TF-IDF, or word embeddings) map semantically related tokens to the same feature index, reducing vocabulary sparsity.
Let's analyze the incorrect options: - Option A is incorrect because dataset splitting (training vs. validation) is an overall machine learning workflow step that occurs at the dataset level, not a step within the text token preprocessing sequence. - Option C is incorrect because saving (serializing) model weights to disk occurs after model training, which is at the very end of the development lifecycle. - Option D is incorrect because making predictions (model inference) occurs after the model has been fully trained and all preprocessing and feature extractions are complete.