A product team runs a multi-agent research-and-draft pipeline in Microsoft Foundry. They want a continuous improvement loop that scores draft quality on faithfulness and structure at scale, then feeds weak cases into prompt and tool refinements. Human labels exist only for a small golden set. Which approach best meets these goals?
Select an answer to reveal the explanation.
Short Explanation
The correct answer is B. You need scale plus a feedback loop, and that is exactly what LLM-as-a-judge plus synthetic hard cases plus golden-set validation gives you. The judge scores faithfulness and structure against a rubric on lots of drafts where no human wrote a ground-truth essay. Patterns from weak scores seed synthetic edge cases so the next prompt and tool changes are aimed at real failure modes, not random tweaks. The small golden set stays the human-trusted gate before you promote changes. Monthly review of ten transcripts is too thin to drive continuous improvement. Bigger models do not remove the need to measure. Turning tools off changes the product under test so you stop evaluating the real multi-agent pipeline.
Full Explanation
Option B is correct. Continuous improvement for multi-agent systems typically combines automated evaluation (LLM-as-a-judge against rubrics for attributes such as faithfulness and structure), synthetic data generation from observed failure modes, and controlled promotion against a small human-labeled golden set. That design scales beyond scarce human labels while still anchoring releases to trusted ground truth.
Option A is incorrect because a monthly sample of ten transcripts cannot provide timely or statistically meaningful coverage for production multi-agent quality, and it provides no systematic mechanism to generate targeted hard cases or validate prompt/tool changes.
Option C is incorrect because model size is not a substitute for evaluation. Larger models can still produce unfaithful or poorly structured drafts, and without measurement the team cannot detect regressions or know whether improvements worked.
Option D is incorrect because the production system uses tools; evaluating pure generation alone ignores tool-result faithfulness and tool-error paths that dominate multi-agent quality risk. Removing tools also invalidates the evaluation relative to the deployed architecture.