A team is building a sentiment classifier from raw customer product reviews. Before feeding the text into any embedding or vectorization step, what preprocessing is most commonly applied first?
Select an answer to reveal the explanation.
Short Explanation
Before a model can do anything useful with review text, that text has to be broken down and normalized. That's tokenization — splitting sentences into words or subword units — paired with lowercasing so 'Great' and 'great' aren't treated as two totally different tokens. That combo is the standard first move in almost every NLP pipeline, which makes it the right answer here. PCA is a dimensionality-reduction trick for numeric feature matrices, not raw characters — it doesn't belong at this stage. One-hot encoding a whole review as a single category throws away all the granular word-level information you actually need for sentiment. And min-max scaling word lengths is measuring the wrong thing entirely — length isn't what tells you if a review is positive or negative. Tokenize and lowercase first, vectorize second.
Full Explanation
Tokenization and lowercasing form the foundational preprocessing step for nearly any text-based deep learning task, including sentiment classification on product reviews. Tokenization splits raw text into discrete units — words, subwords, or characters — that can subsequently be mapped to numerical representations (via a vocabulary index, embedding layer, or subword tokenizer such as WordPiece or BPE). Lowercasing normalizes case so that semantically identical words like 'Great' and 'great' collapse into the same token instead of being treated as distinct vocabulary entries, which would otherwise inflate the vocabulary size and dilute the signal the model can learn from each word form. This step precedes any embedding, TF-IDF, or bag-of-words vectorization.
Principal component analysis is a linear dimensionality-reduction technique applied to numeric feature matrices to find directions of maximum variance; it is not a text preprocessing step and cannot operate meaningfully on raw, unvectorized characters or strings.
One-hot encoding the entire review as a single category is a misapplication of an encoding technique meant for discrete categorical variables with a small number of possible values (like a color or a country). Treating an entire free-text review as one categorical value would discard virtually all information about the words it contains, making sentiment classification impossible.
Min-max scaling of word lengths targets an irrelevant numeric feature. While review length could theoretically be one engineered feature among many in a broader feature set, it is not a substitute for processing the actual textual content, and scaling it does nothing to prepare the words themselves for a model.
The underlying principle: NLP pipelines convert unstructured text into structured, numeric input through a sequence of steps — clean and normalize (tokenize, lowercase, sometimes remove punctuation/stopwords), then vectorize (embeddings, TF-IDF, or one-hot per token), then feed into the model. Tokenization and lowercasing sit at the very start of that sequence.