Public-health chemical names explode a generic tokenizer into many pieces. What tokenizer change fits specialized civic terminology?
Select an answer to reveal the explanation.
Short Explanation
When chemical names shatter into scrap tokens, the clinic notes lose meaning. Grow the WordPiece vocab with domain merges so those terms stick together.
Full Explanation
Public-health clinic notes often contain chemical and medication names that a generic web tokenizer splits into many subword scraps, raising fertility and weakening the signal that links exposures, labs, and treatment language. Expanding the vocabulary with domain WordPiece merges for common chemical names lets those terms stay coherent tokens, improving representation quality for municipal epidemiology and clinical NLP without deleting the terminology the health department must retain. Deleting all chemical mentions from clinic notes before any tokenization removes medically necessary content and biases models away from the very entities public-health workflows need to detect. Switching to hashing tokens randomly so chemicals never share embeddings prevents the model from learning stable chemical identity across notes; random hashes destroy semantic clustering and make rare-but-critical substance names unpredictable. Lowering temperature at inference instead of adapting the tokenizer addresses decoding randomness, not tokenization fertility—temperature cannot glue fragmented chemical pieces back into meaningful units at train or represent time. Exam caveat: domain merges should be driven by measured frequency in the civic corpus and constrained by embedding memory; blindly adding every IUPAC string can bloat vocab without helping rare one-off OCR noise. Operational check: compute fertility and exact-match token recovery on a curated list of clinic chemical strings before and after vocab expansion, and verify GPU embedding memory still fits the clinic inference SKU.