A municipal snow-route incident note is split into sub-words and punctuation before the model processes it. Which LLM basic does this describe?
Select an answer to reveal the explanation.
Short Explanation
Chopping a loaf before toasting is not the same as weighing the slices or sizing the toaster. Tokenization splits text into processable units. It is not the meaning vector and not the window that limits how much text still fits.
Full Explanation
Tokenization is the LLM basic that breaks raw input into tokens—characters, sub-words, words, and punctuation—before further processing. Splitting a municipal snow-route incident note into those pieces is exactly tokenization. Civic testers pasting logs, CAD notes, and route narratives need this first mechanical step kept distinct from meaning vectors and window limits, because troubleshooting starts by knowing what was chopped versus what was budgeted or embedded.
Embeddings fail this definition because they place meanings in a numeric space after tokens exist; proximity of similar ideas is not the chopping step. The context window fails because it limits how much prior text still fits measured in tokens; it does not perform the split. Instruction tuning fails because aligning a foundation model to follow stated test instructions is later model adaptation, not the input chopping step itself.
Exam caveat: tokenization ≠ embeddings ≠ context window. Mixing them causes wrong diagnoses when pasted civic logs behave oddly or when teams blame “the window” for a preprocessing confusion.
Operational check: when a snow-route paste misbehaves, first ask whether input was tokenized as expected, then whether meaning similarity (embeddings) or token budget (window) is the issue; keep the three checks separate in the municipal troubleshooting note and update the paste playbook accordingly.