Municipal code is packed with section numbers like '12.04.030'. How should the team optimize tokenization for those civic identifiers?
Select an answer to reveal the explanation.
Short Explanation
You can keep civic citations useful by training BPE to preserve ordinance numbers like 12.04.030 as rare tokens. That way, they remain recognizable identifiers instead of being split into mystery fragments.
Full Explanation
Ordinance section numbers such as 12.04.030 are stable civic identifiers that clerks, planners, and retrieval systems use to connect questions with the controlling code section. Training BPE to preserve these patterns as rare tokens keeps section IDs coherent in the embedding table, supporting matching, citation generation, and ordinance QA. Replacing every section number with a generic UNK token erases distinctions among ordinances, collapsing them into one unknown bucket and preventing the model from learning which section answers which question. Forcing character-level tokenization of all city text over-fragments ordinary English and Spanish, increases sequence length, and uses more context and GPU memory without providing a production-friendly solution to identifier stability. Enlarging the tokenizer only with emoji merges leaves legal citations fragmented because those merges do not address dotted numeric patterns and spend vocabulary capacity on irrelevant symbols. Exam caveat: preserving rare tokens does not justify unlimited vocabulary growth; measure embedding memory and selectively retain high-value civic patterns. Practical check: tokenize a fixed gold set of ordinance citations, verify each maps to one or a stable few token IDs rather than a long character sequence, and confirm permit/code retrieval F1 does not regress.