A city agency classifies constituent emails. Before any counts or embeddings are built, a prep step cuts each email into the words and subwords the model will actually consume. What is that prep step doing?
Select an answer to reveal the explanation.
Short Explanation
You cannot count the pieces until someone cuts the loaf. You split each constituent email into the words and subwords the classifier will later count or embed. The trap is confusing that cut with stemming, stop-word removal, or a sentiment score.
Full Explanation
Tokenization splits raw text into the units a later model will count or embed. Those units may be words or subwords, but the step's job is to decide the pieces, not to score the email and not to throw pieces away. The agency runs this cut before any counts or embeddings exist, which is the right moment: counts and embeddings are functions of tokens, so the tokens have to come first. Stop-word removal drops common tokens after the token inventory exists. It is a filter on tokens, not the act of creating them. The stem's step cuts the email into the units that will be consumed, including whatever common words the model still keeps. Stemming maps inflected forms onto a root. That changes tokens that already exist. It does not define the original split into consumable units, and subword pieces are not the same thing as linguistic stems. Sentiment scoring assigns a polarity to text. Cutting text into units does not, by itself, decide that an email is positive or negative. A classifier might later use the tokens to score sentiment, but the prep step described is only the split. Exam caveat: tokenizers disagree on punctuation, compounds, and subword boundaries, so two pipelines can count different units from the same email. Operational check: print the token list for one email and confirm those strings are exactly the inputs the count or embedding step receives.