A municipal tree-trimming work-order dump is full of long hyphenated IDs that split into many tokens. What tester implication of tokenization does this illustrate?
Select an answer to reveal the explanation.
Short Explanation
Dense packing peanuts fill a box faster than soft sweaters. Hyphenated IDs, punctuation, and tables burn tokens quickly. Input shape drives how much of a dump still fits in the window the model can see.
Full Explanation
Tokenization implies that input shape—long hyphenated IDs, punctuation, and tables—changes token count and therefore what still fits in the context window. A municipal tree-trimming work-order dump full of dense hyphenated identifiers that split into many tokens illustrates that civic export shape burns budget quickly. Testers designing pastes must treat shape as a first-class fit concern, not an afterthought.
Claims that hyphenated IDs never affect the window because punctuation is free fail—punctuation and ID fragments become tokens. Claims that tokenization applies only after the model has already generated the full answer reverse the processing order. Claims that long IDs prove the model is symbolic AI rather than an LLM misread tokenization density as a type discriminator.
Exam caveat: artefact shape is a token-budget concern on the syllabus. Tables and hyphenated work-order keys are common municipal export hazards when dumping into prompts.
Operational check: sample token expansion on a short work-order excerpt with hyphenated IDs; replace raw ID forests with shortened or hashed keys in the prompt when policy allows; confirm more of the narrative dump remains in-window; note the shape lesson in the tree-trimming civic paste guide.