A city-hall records desk is building an LLM help corpus from ordinance PDFs. Before anyone trains, they need a column of token counts per page so they can drop empty pages and flag oversized ones. What is token length in that step?
Select an answer to reveal the explanation.
Short Explanation
Token counts per ordinance page are a prepared corpus statistic: drop empty pages, flag oversized ones. That is not a Domain 1 token lecture, not tensor padding in app code, and not a Guardrails policy.
Full Explanation
Token length here is a data feature: a column you compute and plot so empty pages drop and monster pages get flagged. It is not a token-definition lecture, not tensor padding in app code, and not a trust policy. Domain 4 owns that corpus statistic before anyone trains.