Orion Commerce has built a Copilot Studio agent to handle customer inquiries for their e-commerce platform covering shipping, returns, product availability, and account issues. The AI team lead, Fatima, is designing the initial test set and wants to ensure it is representative enough to catch real-world agent failures before go-live. She asks a consultant to review her draft test set of 8 questions, all of which ask about shipping timelines in nearly identical phrasing. What should the consultant advise Fatima about test set design for a representative evaluation?
Select an answer to reveal the explanation.
Short Explanation and Infographic
A test set of 8 nearly identical questions is like a mechanic who only checks the engine oil and declares the whole car roadworthy — you'll miss transmission, brakes, tires, and everything else. A representative test set needs breadth across topics, depth in phrasing variety, and enough volume (typically 20-50+) to make evaluation scores statistically meaningful. Correct answer: B.
Full explanation below image
Full Explanation
A representative test set for an AI agent evaluation must satisfy three properties: coverage (it spans all major topic areas the agent handles), variety (it includes diverse phrasings, user personas, and complexity levels), and sufficient volume (enough examples that evaluation scores are statistically reliable rather than coincidental).
Fatima's draft fails on all three counts. With 8 questions that all ask about shipping in nearly identical phrasing, the test set has no coverage of returns, product availability, or account issues — three of the four stated topic areas. It also has no phrasing variety (same sentence structure repeated) and no scenario complexity range (no edge cases, no adversarial inputs, no ambiguous questions).
Option B is correct because best practices for Copilot Studio evaluation recommend a minimum of 20-50 questions as a baseline, with coverage proportional to the agent's topic surface area. A four-domain agent like Orion Commerce's should have test cases in each domain. Variety in phrasing is essential because agents can fail on reformulated questions even when they handle the standard form correctly. Complexity variety is important because agents that pass on typical questions may fail on edge cases.
Option A is incorrect for two reasons: 8 questions is below the recommended minimum, and topic coverage is completely absent for three of four domains. Even if shipping is the most frequent inquiry, an agent evaluation with only one topic assessed cannot be called representative.
Option C is incorrect because artificially limiting the test set to 5 questions to save time defeats the evaluation's purpose. Evaluation is a quality gate, not a checkbox — sparse test sets generate unreliable scores that provide false confidence.
Option D is incorrect because uniqueness of questions is necessary but not sufficient. A test set of 8 unique questions that all cover only one topic still fails the coverage requirement. Size and topic distribution both matter.
Exam tip: The AB-620 exam frequently tests whether candidates understand the difference between a minimal test set and a representative one. Keywords to watch for: 'variety', 'coverage', 'phrasing diversity', and 'minimum 20-50 cases'. When a scenario shows a small, narrow test set, the correct answer nearly always involves expanding it.