Experimentation

NCA-GENL · 66 questions

• A city archives desk wants a language model to draft finding-aid blurbs for newly accessioned boxes. A volunteer is already launching overnight adaptation jobs just to see what happens. What must be written down before the first run?
• A volunteer firehouse tags incident run reports. One overnight job changes the instruction set, the learning rate, and the base checkpoint together, and the crew cannot say which change helped. What is the design fault?
• A county floodplain desk tunes on last spring’s permit notes, peeks at a reserved autumn set after every epoch to pick a winner, and then reports that autumn score as the official result. Why is that score no longer a valid unseen test?
• A community-theater desk claims a customized model writes better playbill synopses but never scored the unmodified pretrained checkpoint on the same gold synopses. What is missing from the experiment?
• A municipal bike-share desk cannot replay last Tuesday’s best adaptation: two new tickets were added to the spreadsheet, the framework was upgraded, and nobody wrote down the random seed. What should have been frozen for a replayable run?
• A public-pool desk has a dozen adaptation attempts sitting in chat screenshots and cannot match which learning rate produced which score. What should they keep instead?
• A farmers-market inspection desk watches training loss fall for many extra epochs while the validation score has already peaked and begun to worsen. What should stop the run?
• A town clerk has one GPU evening and a wish list: full-weight training of a huge model, a small PEFT run, and a prompt-only trial. How should the evening be designed?
• A historical-society minute-book project prints a final number but never checks whether validation tracked training or diverged after epoch two. What should they read as the primary experiment instrument?
• A parks-permit desk declares model B the winner, but A was scored on last year’s questions with greedy decoding and B on this year’s questions with a looser sampler. What is wrong with that comparison?
• A food-bank intake desk has a few hundred labeled comments. One volunteer wants to train a transformer from scratch so it will know pantries; another wants to start from a general pretrained checkpoint and measure the difference on the same split. What is the fair experiment?
• A building-permit desk reuses a general encoder for work-order tags and cannot decide whether to keep the backbone frozen and train only a small head, or to unfreeze later layers. How should they decide?
• A trail-maintenance desk can start from a checkpoint trained on outdoor-incident notes or from one trained on restaurant reviews. Both are called pretrained. Which source is the better first transfer experiment, and why?
• A drawbridge office adapts a general model to bridge-status phrasing and then finds it can no longer answer the everyday weather questions it used to handle. What experimental metric is missing?
• A zoo script desk keeps adding labeled exhibit notes and wonders when training from scratch might catch a transferred model. How should they find out?
• A public-library bindery already agreed to start from a pretrained checkpoint. The remaining argument is whether to update a small PEFT adapter or the full weights on a modest GPU night. How should that argument be settled?
• A county-fair volunteer-crew worked-with graph grew from a classroom clique to tens of thousands of edges. The team wants the same shortest-path and importance queries they already trust. How should they design the runtime experiment?
• A bike-share desk wants to know whether a new who-rebalanced-whom graph recipe still surfaces the same key hubs as last season. What should be the experimental outcome?
• A watershed desk can feed an LLM either a flat list of sensor IDs or a short description of which sensors share a parent tributary — a graph-derived feature. They want to know which input helps a downstream summary. How should they compare those inputs?
• A parking-garage clerk has independent daily occupancy totals and no relationships among the rows. A consultant insists on cuGraph because NVIDIA. What should the clerk do?
• A licensing desk has a small labeled set of permit-status notes. One camp wants a compact encoder classifier; the other wants a decoder that writes the status in prose. How should they compare the two families?
• A parks desk extracts trail-hazard mentions from ranger notes. Staff can freeze the pretrained transformer and train a light head, or they can update more of the stack. What should that experiment measure?
• A planning packet is long, and a short token limit quietly drops the rezoning clause the label depends on. What factor belongs in the experiment, not in hidden preprocessing?
• A 311 desk can paste three graded tickets into the prompt or spend a night on a small supervised adaptation. How should they compare those two treatments?
• A clerk's office loves its note-tagging checkpoint and wants to reuse it unchanged to write inspection summaries. What should the experiment expect?
• A fire-inspection desk needs token tags for code faults. Full-weight training will not fit tonight's GPU; a parameter-efficient adapter will. What comparison should they run?
• A school-lunch allergen classifier won on 120 notes after one lucky shuffle. What should happen before the kitchen changes process?
• A transit delay model looks strong on last April's depot notes and fails on this January's ice-storm notes from another yard. What eval belongs in the reported experiment?
• A records clerk must point at the filing date already written in a short passage. One score requires the span to match gold characters exactly; another gives partial credit when the words mostly overlap. How should they pick?
• A water desk scores extractive 'where is valve 9?' answers with a translation-style n-gram overlap number and crowns a chatty restatement. What is wrong with that primary metric?
• A rec-center QA set accidentally includes the same 'what is the pool fee?' items that were pasted as few-shot hints in the prompt. Why is the comparison contaminated?
• A public-works office wants to know whether this week's checkpoint beats last month's on 'which pump tripped?' What makes that head-to-head valid?
• A parcel-lookup helper has a high automatic QA score, but clerks say it names the wrong lot even when a few words overlap the gold span. What should the experiment add?
• A complaint archive uses a second language model to grade long answers about who filed which case. The judge prefers florid answers that match its own style. What is that judge good for at associate depth?
• A bridge-closure question needs a two-sentence explanation of why the span closed, not a three-word date. How should the models be ranked?
• A permit-desk QA bake-off used sampling; model A won by two questions on a 40-item set. What should happen before they retire model B?
• A street-sign project tags asset names. Most tokens are ordinary words, so a model that marks everything other looks accurate. Which score should they trust?
• A housing desk classifies emergency versus routine notes. Routine notes dominate, and a model that always predicts routine looks accurate while missing every emergency. Which metric surfaces the rare class?
• A library-event bulletin summarizer wins a reference-overlap contest by copying sentences, while readers prefer a shorter rewrite that a human panel ranks higher. What do those two scores mean?
• A flood desk brags that an adapted model has lower perplexity on river chatter, then ships it as a flood / no-flood classifier without scoring those labels. What did they mix up?
• A city clerk must recast each parking-appeal ruling into a statute-mandated one-sentence formula that already has a gold restatement. Which automatic metric is a reasonable primary score for that constrained rewrite, yet a weak primary for open-ended chat?
• A public-library help desk lets one intern star thirty catalog-note summaries and calls the stack human evaluation. What must a real human-eval experiment include?
• A county elections office pulls one pretrained LLM and wants four jobs: token tags on polling-place names, a label for each precinct note, a short recap of a canvass memo, and a which-site question. How should they score the run?
• A municipal water desk's automatic scores say a new summarizer is better; night operators say it drops boil-water alerts. What should the experiment do?
• A city parks reservation desk has two standing instruction sheets for the same LangChain job. How should they compare those sheets?
• A public-works snow-route desk keeps the LangChain document-analysis path identical and only changes which LLM sits behind it. What is the experimental factor?
• A municipal archives desk can stuff a week of council minutes into one call, or map over each day's minutes and then reduce. How should they treat those two shapes?
• A transit lost-and-found desk tries the chain in a chat window whenever someone feels like it. What turns that habit into an experiment?
• A building-permits counter improved after someone quietly changed the template and the model name in the same afternoon. What must a scored chain experiment record?
• A recycling-center desk rewrites LangChain calls into cleaner objects, keeps the model and prompts the same, and wants to publish a quality win. How should that change be labeled?
• A public-housing waitlist office finds two Hub models tagged for the same kind of note classification. What is the right comparison?
• A zoning-board intern grabs the most-downloaded Hub checkpoint even though the model card describes a different language and task than their gazetteer notes. What should decide eligibility for the run?
• A tax-assessor team fine-tunes on a Hub dataset whose card warns that train and test share near-duplicate parcels. What should they do about the published test split?
• A fire-inspection loft wants to experiment by putting each candidate checkpoint behind a live server after every epoch. Where does the comparison belong?
• A municipal print-shop lab finishes a parameter-efficient run and tries to check in a full copy of the base model as the result. What should the run record as output?
• A flood-alert desk says they used the Hub summarizer last month; the default revision moved and this month's rerun does not match. What must a versioned run log include?
• A public-health inspection desk has a mid-size instruction set and one workstation GPU. Full-weight supervised fine-tuning would update every parameter; LoRA would freeze the base and train low-rank adapters. Which experiment should they run first, and why?
• A community-garden seed library raises adapter rank so the model can learn more patterns on a few dozen instruction cards. Training loss collapses while validation falls apart. How should rank be treated?
• A courthouse records office wants to adapt a larger pretrained model that will not fit in memory for ordinary LoRA. Which associate-depth choice lets the experiment run?
• A school-cafeteria menu desk needs one base model that can tag allergen notes or draft tray labels without training a second full copy. How can a multi-task experiment do that?
• A city band-shell programming desk has very little labeled program-note text and can either learn soft prompt or prefix parameters or attach a LoRA-style weight adapter. How should those two PEFT families be framed?
• A voter-registration desk can scrape two thousand noisy instruction pairs or carefully edit two hundred. A consultant says large language models just want scale. How should they design the supervised-adaptation data experiment?
• A historical-society desk writes eighteen precious question-and-answer cards, trains until the model recites them, then fails on a rephrased nineteenth card. What experimental response fits this instruction-tuning overfit?
• A municipal millpond office copies a pretraining-scale learning rate and a long epoch count into a LoRA run on a short instruction list. Validation dies by dinner. What should change?
• A city-council clerk wants replies that stay helpful and on-register. They can supervised-fine-tune on gold replies, run a human-feedback alignment experiment, or use SteerLM-style attribute control at inference. How should they choose?
• A parking-meter office must decide tonight's work: rewrite the standing instructions, run a LoRA-style adaptation on a small gold set, or retrieve the latest rate tables at ask-time. Facts change weekly; tone should sound like the traffic commissioner. Which experiment, or order of experiments, should they run, and which stack customizes a model?