Data Preparation

NCP-GENL · 18 questions

• 311 ticket exports used for triage fine-tuning are missing many neighborhood fields, but latitude and longitude are usually present. What should curators do first in data preparation?
• Inspection severity labels for a classifier LLM are about 90% pass, so rare failures barely appear in training batches. Which curation step best addresses that imbalance?
• Public-health clinic notes vary from a few words to multi-page dumps with noisy whitespace. What cleaning step should analysts take before packing sequences for an LLM?
• Transit delay-reason codes look skewed, and staff suspect weather labels dominate training batches. What should the team do before model training?
• Assessor deed OCR exports include many null grantor names that would poison supervised training if left as empty strings. What is the sound data-curation choice?
• Housing complaint categories were renamed across several years, fracturing what should be the same classes. How should curators prepare the labeled set?
• Court-opinion fine-tuning requires JSONL with prompt and completion fields, but staff only have mixed DOCX dumps. What organization step is required?
• A parks chatbot dataset includes summer festival chatter and winter program text. How should the team split train, validation, and test to avoid leakage?
• Water SCADA narrative logs must be ingested by a NeMo-style pipeline that expects clean text records. Which format preparation is appropriate?
• Library catalog blurbs must be paired with subject headings for supervised tuning. How should the dataset be structured?
• Emergency after-action PDFs will feed both retrieval and fine-tuning. What organization practice keeps those sets aligned?
• County grant applications arrive as scanned TIFF images that must become modeling text. Which preparation step is appropriate?
• Municipal code is packed with section numbers like '12.04.030'. How should the team optimize tokenization for those civic identifiers?
• A bilingual city must choose WordPiece versus BPE vocabulary sizes for English–Spanish permitting forms under tight GPU memory caps. What trade-off should lead the decision?
• Public-health chemical names explode a generic tokenizer into many pieces. What tokenizer change fits specialized civic terminology?
• Transit GTFS stop names tokenize into too many pieces. What should analysts measure and retune?
• School-district IEP notes must run on small edge inference boxes in nurse offices. How should the tokenizer vocabulary be sized?
• Tax form checkboxes become unknown tokens during training. What should the team do before modeling?