A community wind-ensemble helper scores high on a public summarization benchmark but fails on program-note house style. What should the team add?
Select an answer to reveal the explanation.
Short Explanation
Think of a helper that looks great on a public summarization quiz but still writes the wrong program-note voice. Add a task-specific set. A generic high score, no eval, or a content filter does not prove house style.
Full Explanation
A community wind-ensemble helper that scores high on a public summarization benchmark can still fail program-note house style because public benchmarks measure a generic task, not the business task the ensemble actually ships to concert audiences. Adding a task-specific evaluation set works because it scores the outputs that matter—voice, structure, and program-note conventions for concert programs—rather than a generic summary score. Trusting the public benchmark alone ignores the house-style failure mode already observed in production-like drafts. Dropping evaluation because a generic score looks high removes the feedback loop that would catch style regressions after prompt or model changes. A content filter blocks disallowed content; it does not measure house style or program-note formatting for the ensemble. Exam caveat: task-specific sets must stay fresh as house style evolves, and they should include hard edge cases, not only easy happy-path notes. Operational check: build a labeled program-note evaluation set, score the current helper against it, treat house-style pass rate—not the public benchmark—as the promotion gate, and re-run that set after every material change to the helper.