The Copilot Studio development team at Contoso has made significant changes to their agent's generative AI configuration and knowledge sources. Before promoting the new version to production, they want to objectively compare the response quality of the new version against the current production version. What is the recommended approach?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Comparing agent versions objectively is like A/B testing in software — you run both against the same test suite and let the metrics decide. Batch evaluation against a golden dataset gives you comparable, numerical quality scores. The correct answer is B.
Full explanation below image
Full Explanation
## Why B is Correct Batch evaluation runs a predefined set of test questions (the 'golden dataset') against both versions of the agent and generates evaluation metrics (Precision, Recall, Groundedness, Relevance, Coherence). Comparing these metrics across versions gives an objective, quantified view of whether the new version represents a quality improvement. This is the standard method for pre-deployment validation of generative AI changes.
## Why the Distractors Are Wrong A (10% production traffic canary): Canary releases are valid for incremental rollout, but they expose real customers to a potentially degraded experience. This approach should come after pre-production validation, not instead of it.
C (Manual review by 5 team members): Manual evaluation with 10 questions each is too small a sample, subject to human bias, and not reproducible. It provides qualitative impressions, not objective metrics.
D (New knowledge = always better): New knowledge sources can improve recall but may also introduce inconsistencies, change response tone, or regress previously working scenarios. Assumption without testing is a production risk.
## Exam Tip Pre-deployment quality validation = batch evaluation against golden dataset. This is the platform-native, objective method. Know that golden datasets are curated Q&A pairs with known correct answers used as a benchmark.