A housing eligibility prompt confuses income thresholds on eval. What refinement practice fits?
Select an answer to reveal the explanation.
Short Explanation
If eval says income thresholds are muddy, tighten the prompt. Don't just add more income cases while keeping the same fuzzy threshold language; it won't fix the cutoff confusion.
Full Explanation
Prompt refinement after eval exposes income-threshold confusion is how housing eligibility assistants improve without guessing. Iterating the housing eligibility prompt when tests show threshold confusion works because eval failures are concrete evidence that Claude is misreading cutoffs, household composition rules, or documentation exceptions that determine resident outcomes. Targeted edits—clearer threshold tables, worked examples, forbidden invention rules—directly attack the measured failure mode instead of hoping production traffic will self-correct.
Monitoring eval results quarterly and updating thresholds only when patterns emerge fails because a known confusion flagged today should not sit unaddressed for a quarter in the path that advises applicants about benefits and waitlists. Simplifying the income threshold rules and letting Claude infer boundary cutoffs fails because removing explicit cutoffs invites Claude to guess program policy at the exact boundaries where eval already showed it gets confused. Expanding the eval dataset with additional income cases while retaining the current threshold language fails because more cases cannot fix unclear cutoff wording in the prompt; it only reruns the same confusion across extra inputs instead of giving applicants clearer thresholds.
Exam caveat: iterating on one failing dimension can regress another (for example, fixing income tables while breaking household-size logic), so changes need full regression, not single-case patches. Operational check: log the failing eval cases, revise threshold language with program staff, re-run the eligibility suite including edge incomes near cutoffs, and only promote the prompt when threshold accuracy meets the housing program's acceptance bar.