Orion Commerce has built a Copilot Studio agent to handle customer inquiries for their e-commerce platform covering shipping, returns, product availability, and account issues. The AI team lead, Fatima, is designing the initial test set and wants to ensure it is representative enough to catch real-world agent failures before go-live. She asks a consultant to review her draft test set of 8 questions, all of which ask about shipping timelines in nearly identical phrasing. What should the consultant advise Fatima about test set design for a representative evaluation?

Representative test sets for AI agent evaluation must satisfy three properties: coverage (they span all major topic areas the agent handles), variety (they include diverse phrasings, user personas, and complexity levels), and sufficient volume (enough examples that evaluation scores are statistically reliable rather than coincidental). Fatima's draft fails on all three counts. With 8 questions that all ask about shipping in nearly identical phrasing, the test set has no coverage of returns, product availability, or account issues — three of the four stated topic areas. It also has no phrasing variety (same sentence structure repeated) and no scenario complexity range (no edge cases, no adversarial inputs, no ambiguous questions). The recommendation to include minimum of 20-50 questions covering all major topic areas, with varied phrasing, user types, and scenario complexity, is correct because best practices for Copilot Studio evaluation recommend minimum of 20-50 questions as baseline, with coverage proportional to the agent's topic surface area. Four-domain agents like Orion Commerce's should have test cases in each domain. Variety in phrasing is essential because agents can fail on reformulated questions even when they handle the standard form correctly. Complexity variety is important because agents that pass on typical questions may fail on edge cases. The claim that the test set is adequate because shipping is the most common inquiry category and 8 questions is sufficient for initial evaluation is incorrect for two reasons: 8 questions is below recommended minimum, and topic coverage is completely absent for three of four domains. Even if shipping is the most frequent inquiry, agent evaluation with only one topic assessed cannot be called representative. The suggestion to limit the test set to 5 questions to keep evaluation fast, focusing only on the highest-priority scenario, is incorrect because artificially limiting the test set to 5 questions to save time defeats the evaluation's purpose. Evaluation is quality gate, not checkbox — sparse test sets generate unreliable scores that provide false confidence. The claim that test set size does not matter as long as each question is unique, and that 8 questions is valid baseline regardless of topic coverage, is incorrect because uniqueness of questions is necessary but not sufficient. Eight unique questions that all cover only one topic still fail the coverage requirement. Size and topic distribution both matter. Exam tip: The AB-620 exam frequently tests whether candidates understand the difference between minimal test set and representative one. Keywords to watch for: 'variety', 'coverage', 'phrasing diversity', and 'minimum 20-50 cases'. When scenario shows small, narrow test set, correct answer nearly always involves expanding it.