A product team runs a multi-agent claims workflow in Microsoft Foundry. Human spot-checks cannot keep up with nightly regression volume. They need an automated evaluation loop that scores agent final answers for faithfulness to retrieved knowledge and for policy compliance, then flags regressions before promotion. Which approach best meets this continuous-improvement need?
Select an answer to reveal the explanation.
Short Explanation
Correct answer: A. When human review cannot scale, you still need structured quality signals before you promote a multi-agent build. An LLM-as-a-judge loop scores each run against clear rubrics—faithfulness to retrieved knowledge and policy compliance—then ties those scores to run metadata so you can trend regressions. Promotion gates that fail on score drops catch drift early. Exact string unit tests (B) break as soon as phrasing changes and miss semantic failures. Cranking temperature and running one golden prompt (C) is not continuous evaluation. Turning off tools (D) evaluates a different system than production. Keep humans for high-risk edge cases; automate the bulk scoring.
Full Explanation
The correct option implements continuous improvement with an LLM-as-a-judge framework: automated rubrics for faithfulness and policy compliance, persisted scores, and a hard promotion gate on regression. This aligns with Foundry-oriented evaluation practice for multi-agent solutions where volume exceeds pure human review. Option B is incorrect because exact string assertions do not tolerate valid paraphrase and fail to detect semantic hallucinations or policy violations. Option C is incorrect because changing temperature and sampling a single prompt weekly does not provide systematic, comparable evaluation across workflows. Option D is incorrect because disabling tools evaluates a reduced system that does not represent production tool-using agents. A practical design pairs automated judges with human review on stratified samples and stores correlation IDs so failed runs can be replayed.