A county evaluation team debates whether their 40-prompt golden dataset is large enough to draw a reliable conclusion before promoting a new permitting-agent version. What consideration should weigh most heavily on whether 40 prompts is enough for this go/no-go decision?
Select an answer to reveal the explanation.
Short Explanation
A handful of test questions can make an agent look great or terrible almost by chance, the same way judging a restaurant off one dish tells you less than eating there a dozen times. What actually matters isn't the number 40 itself, but whether those 40 prompts cover the real variety of situations the agent will hit in production.
Full Explanation
Sample-size sufficiency for a golden dataset isn't about hitting a specific count, it's about whether the set spans the distribution of real scenarios, including edge cases and failure-prone categories, with enough prompts per category that a single lucky or unlucky run doesn't swing the overall score unrepresentatively; a small, narrow set can produce a confident-looking pass or fail that doesn't generalize to production traffic. Matching a prior unrelated evaluation's prompt count is coincidental and says nothing about whether this particular agent's scenario space is adequately covered. Choosing 40 because it presents cleanly in a report optimizes for presentation over statistical reliability, which is the opposite of what a go/no-go gate should prioritize. Framework default configurations may exist for convenience, but a tool accepting a given prompt count without erroring is not evidence that the count is statistically meaningful for the decision being made. Scope caveat: even a well-distributed sample can still miss rare but high-impact edge cases, so dataset size should scale with how consequential a missed scenario would be for permitting decisions. Operational check: break the 40 prompts down by scenario category and confirm each category the agent will face in production has more than a token one or two examples before trusting the aggregate score.