In a natural language processing (NLP) pipeline, raw text must be processed by a tokenizer before it is fed into a model. What is the definition of a "token" in this pipeline?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: computers don't read books or articles the way you and I do. They can't process a raw string of characters like "Cisco routers are fast" without breaking it down first. Think of a tokenizer like a digital paper shredder. You feed it a sentence, and it cuts it up into standard little pieces. Those pieces are called tokens. Now, depending on the tokenizer, a token might be a whole word like "Cisco", a part of a word (a subword) like "rout" and "ers", or even just a single letter or punctuation mark. Once the text is shredded into these tokens, the model converts them into numbers so it can do its math. Hopefully you chose D, because tokens are the absolute building blocks of any NLP input.
Full explanation below image
![A flow diagram showing tokenization. Raw text "unbelievable!" is shown at the input. It passes through a "Tokenizer" block. At the output, it is split into four distinct boxes representing tokens: ["un", "believable", "!"]. Show how these tokens map to integer IDs (e.g., [124, 8763, 5]) which are then fed into the model's embedding layer.](/media/blockchain-council/caid/CAID-q149-infographic.webp)
Full Explanation
In Natural Language Processing (NLP), tokenization is the foundational step of preprocessing text. A token is defined as the atomic unit of text that a machine learning model operates upon.
Depending on the tokenization strategy employed, tokens can be defined at different granularities: 1. Word-level Tokenization: The text is split by spaces and punctuation, where each word becomes a token. While intuitive, this leads to a vocabulary size and cannot handle out-of-vocabulary (OOV) words or spelling mistakes well. 2. Character-level Tokenization: The text is split into individual letters and punctuation. This results in a very small vocabulary (typically under 100 characters) but makes sequences extremely long and makes it harder for the model to learn word-level semantics. 3. Subword-level Tokenization (Standard for Modern LLMs): Methods like Byte-Pair Encoding (BPE) or WordPiece split common words into whole tokens and rare words into smaller subwords (e.g., "unwanted" might become "un" and "wanted"). This strikes an ideal balance, keeping the vocabulary manageable while easily handling new or misspelled words.
Let's examine why the other options are incorrect: - Option A describes a class label or target variable in classification tasks, not an input unit of text. - Option B describes a word embedding or vector representation, which is what a token is converted into after tokenization, but the token itself is the textual unit. - Option C describes control tokens (like [SEP], [CLS], or <|endoftext|>), which are a specific subset of tokens, but the term "token" refers broadly to any input unit of text.