Public-health clinic notes vary from a few words to multi-page dumps with noisy whitespace. What cleaning step should analysts take before packing sequences for an LLM?
Select an answer to reveal the explanation.
Short Explanation
Wild note lengths are like mixing sticky notes with novels in the same inbox. Clean the whitespace and clip the crazy outliers before you pack sequences for the model.
Full Explanation
Public-health clinic notes that range from a few words to multi-page dumps with noisy whitespace should be cleaned by normalizing whitespace and clipping extreme length outliers before packing sequences for an LLM. Normalization removes control-character and spacing artifacts that inflate token counts without adding clinical meaning; clipping extremes prevents a few mega-notes from dominating packed contexts or blowing sequence budgets. These are foundational cleaning steps for heterogeneous civic clinical text. Keeping raw control characters and never clipping length treats every byte as sacred and preserves noise that harms tokenization and packing efficiency. Concatenating all notes into one unbroken megabyte string with no separators destroys document boundaries and causes cross-note leakage inside packed sequences. Converting every note to a single emoji to minimize token use obliterates clinical content and is not a serious preparation strategy for LLM training or inference on clinic narratives. Exam caveat: clipping outliers should retain pointers to full notes in the source system when legally needed; do not silently destroy records—store truncated training views separately from systems of record. Operational check: measure length percentiles, define clip thresholds, apply whitespace normalization consistently, inspect packed batches for separator integrity, and confirm extreme dumps no longer dominate sequence packing.