Priya is an AI developer at Contoso Financial Services who has built a Copilot Studio agent that answers questions about investment products. Her manager asks her to set up a formal evaluation framework using a test set before the agent goes live. Priya needs to understand what a properly structured test set must contain to enable accurate AI evaluation scoring in Copilot Studio. Which combination of elements is required in each test set entry for evaluation to function correctly?
Select an answer to reveal the explanation.
Short Explanation
Think of a test set like an answer key. You need the question, the ground-truth answer, and the source context. Without all three, Copilot Studio can't score whether the agent's answer was accurate or grounded.
Full Explanation
In Copilot Studio's AI evaluation framework, a test set serves as the authoritative benchmark against which agent responses are measured. Each entry in a valid test set must include three core components: the user query (what the user asks), the expected response — also called ground truth — (what a correct answer looks like), and the relevant context or knowledge source that the agent should draw from.
The combination with a user query, the agent's live response, and a confidence score threshold is incorrect because confidence score thresholds are an output metric, not an input to a test set. You configure threshold tolerances separately in the evaluation settings — they do not form part of the test data itself. Additionally, using the agent's own live response as part of the test set input would create circular evaluation logic.
The combination with a user query, a topic label, and a fallback response for when the agent fails is incorrect because topic labels are organizational metadata used for routing within Copilot Studio topics — they are not an evaluation input. Fallback responses are safety nets for failed interactions, not test set components.
The combination with a user query, a conversation transcript, and a reviewer-assigned star rating is incorrect because conversation transcripts may be useful for retrospective analysis, but they are not a structured evaluation input. Star ratings represent human sentiment and are not how Copilot Studio's AI-based evaluation metrics (groundedness, relevance, coherence) are computed — those require expected responses and context, not reviewer ratings.
The combination with a user query, an expected response (ground truth), and the relevant knowledge source or context is correct because the AI evaluation engine in Copilot Studio compares the agent's generated response against the expected response (ground truth) and checks whether the response is grounded in the provided context. Without the expected response, groundedness and relevance scores cannot be computed meaningfully.
Exam caveat: Remember the triad — query, ground truth, context. If any one is missing, evaluation scoring breaks down. On the exam, watch for choices that mix in live agent output or human ratings, which are outputs, not inputs to the test set.