A municipal snow-route incident note is split into sub-words and punctuation before the model processes it. Which LLM basic does this describe?
Select an answer to reveal the explanation.
Short Explanation
Chopping a loaf before toasting is not the same as weighing the slices or sizing the toaster. Tokenization splits text into processable units. It is not the meaning vector and not the window that limits how much text still fits.
Full Explanation
Tokenization converts raw input into tokens—character, sub-word, or word pieces including punctuation—so the model can process the text. Embeddings then represent meaning numerically; the context window limits how many tokens of prior text remain in view. Mixing those three basics causes testers to misdiagnose why a pasted log behaves oddly. Instruction tuning is a later model-adaptation idea, not the chopping step itself.