Evaluate, optimize, and monitor multi-agent solutions (20–25%)

Microsoft Certified: Multi-Agent AI Solutions Expert · 72 questions

• After deploying a multi-agent customer-email system in Microsoft Foundry, quality fluctuates weekly. You need continuous improvement beyond occasional manual spot checks. Which evaluation approach best fits?
• Operators notice a long-running multi-agent investigation “forgets” earlier entities: names morph, amounts change, and later agents contradict earlier tool results after repeated summarization. Which diagnosis and response is most appropriate?
• Finance flags rising spend on a multi-agent Azure solution. Engineering must monitor reliability and cost: token burn, loop storms, tool-call fan-out, and workflow failures—with actionable alerts. Which strategy best addresses this?
• A product team runs a multi-agent claims workflow in Microsoft Foundry. Human spot-checks cannot keep up with nightly regression volume. They need an automated evaluation loop that scores agent final answers for faithfulness to retrieved knowledge and for policy compliance, then flags regressions before promotion. Which approach best meets this continuous-improvement need?
• After a production incident, engineers must reconstruct why a supervisor agent approved an incorrect subagent tool call. Logs exist in several microservices, but they cannot link one user session across the orchestrator, tool gateway, and model calls. Which observability design should they implement in Microsoft Foundry?
• Users report that a long-running research multi-agent loses early constraints (budget cap and excluded vendors) after many tool turns, then invents conflicting recommendations. Token usage per session is near the model limit. Which diagnosis best explains the failure mode?
• A parallel research swarm repeatedly re-queries the same MCP search tool in tight loops, driving token spend and latency far above SLA. Product wants lower cost without removing the tool entirely. Which optimization strategy should you implement first?
• Over six weeks, a support multi-agent’s automated evaluation suite shows rising policy-violation rates even though code deploys were frozen. Knowledge articles were updated daily by content owners. What monitoring strategy best detects and explains this pattern?
• Finance requires chargebacks for a shared multi-agent platform used by three business units. Today all Foundry usage lands under one subscription with no attribution. What cost-management design should the platform team implement?
• Before promoting a multi-agent mortgage underwriting workflow, compliance requires human review of a statistically valid sample of agent decisions, including tool inputs and final rationales. How should the team design this process in Microsoft Foundry?
• An orchestrator fans out ten research subagents sequentially even though their tasks are independent. End-to-end duration exceeds the five-minute SLA, while the model deployment still has headroom before rate limits. What should you do to optimize task duration?
• A platform squad wants evaluations that isolate failures across memory, knowledge retrieval, tools, and prompts rather than only scoring final answers. Which evaluation design best supports root-cause localization?
• Before promoting a multi-agent workflow in Microsoft Foundry, product owners must score groundedness and task success on a labeled set. Automated metrics alone are not trusted yet. What should you design?
• After a release, users report the research agent forgets case facts mid-session, tools return unused, and prompts drift from the approved template. What evaluation strategy should you implement?
• Your multi-agent batch job finishes too slowly. Profiling shows sequential specialist calls and frequent 429s from the model endpoint. What optimization should you apply first?
• Analysts notice the investigation agent forgets entities introduced early, summaries slowly change key facts, and vector recall sometimes returns topically similar but wrong cases. Which diagnosis best matches these context-window issues?
• You need continuous improvement for a multi-agent support solution: scalable scoring of free-text answers, periodic synthetic hard cases, and incorporation of user thumbs-down. Which approach best fits?
• Operations needs a reliability strategy for multi-agent production: detect unhealthy agents, workflow failures, quality regressions, and behavioral drift, then trigger remediation. What should you design?
• A multi-agent concierge promises 99.9% workflow success within 8 seconds end-to-end. Leadership wants platform and solution monitoring aligned to that SLA. What strategy is appropriate?
• Token spend for a planner-heavy multi-agent graph spiked after agents began long tool-call loops. What optimization strategy should you implement?
• Finance requires showback for multi-agent usage across business units, with quotas and alerts before overage. What monitoring and management approach should you design?
• Debugging a failed multi-agent run is hard because prompts, token counts, and tool calls are scattered without a single timeline. What should you implement in Microsoft Foundry?
• After a prompt change to the triage agent, automated offline scores look stable, but live users report more misroutes. What monitoring gap should you close?
• A rare multi-agent failure occurred once in production. Engineers cannot reproduce it from logs that only store final answers. Which observability capability is most important to add?
• You need a repeatable way for domain experts to score multi-agent runs in Microsoft Foundry before promoting a new orchestrator version. Automated metrics exist but miss subtle policy tone issues. What should you design?
• After a release, overall task success drops. You suspect either degraded memory retrieval, a broken tool contract, or a prompt regression—but a single end-to-end pass/fail metric cannot tell you which. What evaluation approach should you implement?
• A parallel research workflow spawns eight specialist agents that all hit the same model deployment and the same external search API. Median task duration has doubled; logs show frequent 429 responses. What optimization should you apply first?
• A long-running multi-agent investigation repeatedly summarizes its own history to fit the context window. Over time, critical entity attributes change incorrectly in the summary (for example, customer region and contract tier), and later agents act on the wrong facts. Which diagnosis and remediation pair is best?
• You want continuous improvement for a multi-agent support solution: large volumes of traces, limited expert time, and a need to catch semantic regressions between weekly releases. Which approach best fits?
• Operations reports that a multi-agent workflow’s quality slowly worsened over two weeks without a code deploy. Tool schemas are unchanged. You need a monitoring strategy that catches this class of failure and supports response. What should you design?
• A multi-agent booking solution has a contractual SLA: 99.9% successful completion of the booking workflow within 45 seconds for interactive sessions. Platform CPU is healthy, but customers report timeouts when the orchestrator waits on a slow third-party MCP tool. What monitoring design addresses SLA adherence?
• Cost and latency spiked after a prompt change. Traces show agents repeatedly calling the same tool and re-planning in long loops until max tokens. Which optimization strategy should you implement?
• A shared multi-agent Foundry platform serves three business units. Finance demands showback of model and tool spend per unit; one unit is exhausting the shared quota and starving others. What cost-management strategy should you design?
• Debugging a failed multi-agent run requires reconstructing which orchestrator decision led to which tool call and model response across services. Today logs are disconnected per component. What should you implement in Foundry-oriented observability?
• Interactive multi-agent sessions truncate oldest turns when the context window fills. Users report that early constraints (“never email the customer directly”) disappear mid-session and agents violate them later. Which remediation best targets sliding-window amnesia?
• A team updates system prompts for three specialist agents weekly. Last month a prompt edit improved tone but increased incorrect tool arguments. You need an evaluation practice that prevents similar regressions. What should you do?
• Your team must evaluate a multi-agent claims solution in Microsoft Foundry before wider release. Stakeholders want structured human review of agent outcomes, not only automated scores. Which approach best designs the human review process?
• Users report that a long-running multi-agent project assistant forgets earlier constraints after many turns. Engineers notice summaries gradually omit the original budget cap while vector recall returns loosely related notes. Which diagnosis and remediation pairing is best?
• After launch, you need continuous improvement for a multi-agent assistant: detect quality regressions, generate hard test cases, and fold user thumbs-down signals into optimization. Which strategy best fits AI-500 continuous improvement practices?
• A multi-agent claims assistant in Microsoft Foundry keeps a rolling conversation context using a fixed-size sliding window. Over long claims investigations, the policy-number entity and claimant name that were established in the first few turns disappear from later agent prompts, and downstream agents begin inventing identifiers. Offline evaluation shows retrieval still finds the correct claim, but the writing agent no longer mentions the original entities. Which diagnosis best explains the failure mode, and what should the team address first?
• A product team runs a multi-agent research-and-draft pipeline in Microsoft Foundry. They want a continuous improvement loop that scores draft quality on faithfulness and structure at scale, then feeds weak cases into prompt and tool refinements. Human labels exist only for a small golden set. Which approach best meets these goals?
• Tracing in a Foundry multi-agent support solution shows that a troubleshooting agent sometimes issues more than forty tool calls for a single ticket, repeatedly querying similar telemetry endpoints with slightly rephrased parameters. Token spend and latency spike on those tickets even when a useful answer was available after five or six calls. Which optimization best reduces waste while preserving legitimate multi-step diagnosis?
• Contoso runs three business units on a shared multi-agent Foundry platform. Finance complains that monthly Azure OpenAI spend is opaque: one unit's research swarm appears to consume most tokens, but chargebacks are still split evenly. Platform engineering must improve cost accountability without redesigning every agent. What should they implement first?
• A multi-agent project team shares a rolling conversation summary that each specialist agent updates after its turn. Over multi-day projects, the summary gradually invents milestone dates and drops constraints the user stated early on. Individual agent traces look locally coherent. Which evaluation focus best detects this class of failure?
• After a failed multi-agent checkout flow, support engineers can see each microservice log stream but cannot reconstruct which orchestrator decision led which subagent to call which payment tool for that one customer order. What observability design should they implement in Foundry and related services?
• Six weeks after launch, a multi-agent HR policy assistant still passes its original pre-release evaluation suite, but employees report that answers have become more verbose, less actionable, and occasionally cite retired policy versions after knowledge-base updates. Which monitoring strategy best closes this gap?
• An architect plans to cut median latency of a six-agent research workflow by running independent retrieval specialists in parallel. During a pilot, total wall-clock time barely improves and 429 rate-limit errors surge against a shared Azure OpenAI deployment and a shared search service. What should the team do to optimize task duration realistically?
• A regulated insurer must approve any change to a multi-agent underwriting assistant before production. Automated Foundry evaluations score the suite green, but risk stakeholders require evidence that humans reviewed edge cases the metrics might miss. Which process design best satisfies both speed and governance?
• A multi-agent claims team needs continuous quality scoring on thousands of nightly agent runs. Human review can cover only a small sample. They want automated semantic scoring of final answers against policies, with periodic human calibration, feeding a semantic optimization loop. Which approach best fits Foundry-oriented continuous improvement?
• Support engineers debugging a multi-agent failure in Microsoft Foundry can see that a user request failed, but they cannot reconstruct which agent ran, which tools were called, how many tokens were consumed, or how child spans related to the parent request. Which observability implementation should the team prioritize?
• Users report that a long-running multi-agent planning session “forgets” goals stated at the beginning of the conversation after many turns, even though no explicit memory wipe was coded. Engineers confirm the implementation uses a simple last-N message window with no summarization of older turns. Which context-window issue is this, and what is the appropriate diagnostic framing?
• Cost dashboards show one multi-agent workflow consuming extreme token volume because the orchestrator and a critic agent refine the same paragraph for dozens of rounds with no stop condition, each round including full history and multiple tool calls. Which optimization strategy best targets this failure mode?
• After a prompt change to the triage agent, automated offline scores looked fine on a tiny smoke set, but production customer satisfaction for multi-agent resolutions dropped 12% week over week. On-call had no alert because only infrastructure uptime was monitored. What monitoring capability should be added?
• A team must expand evaluation coverage for a multi-agent benefits advisor before open enrollment. Real labeled production transcripts are scarce because of privacy constraints. They need diverse scenarios (edge cases, adversarial phrasings, multi-turn tool paths) to test prompts, tools, and memory. Which continuous-improvement technique best addresses the gap?
• Finance reports that one business unit’s multi-agent experiment consumed the majority of the shared Azure OpenAI quota, starving production agents in other units. Leadership wants usage visibility by team, hard allocations, and chargeback. Which monitoring and management strategy addresses this?
• A multi-agent research workflow currently runs four independent specialist agents one after another; total duration is the sum of all four. Model deployment rate limits still have headroom for higher concurrency. Product requires lower p95 latency without changing answer quality targets. What optimization should the team apply first?
• Before promoting a new multi-agent configuration in Microsoft Foundry, the responsible AI lead requires structured human review of sampled traces: correctness, tool appropriateness, and policy adherence—not only automated scores. Which evaluation practice should the team implement?
• A multi-agent session uses recursive summarization to fit long histories into context. After many cycles, the running summary still “sounds coherent” but has silently changed a key constraint from “exclude vendor X” to “prefer vendor X,” and downstream agents act on the wrong constraint. Which context issue is this, and what is the right response?
• Platform ops needs a reliability strategy for a production multi-agent system: detect when individual agents are unhealthy, when workflows fail mid-orchestration, when traces cannot be correlated, and when automated remediation should page on-call or restart a failed step. Which monitoring strategy scope is most complete?
• A multi-agent release checklist currently only unit-tests Python helpers. Failures in production are often caused by bad tool schemas, weak retrieval, prompt regressions, or memory mis-scoping—not helper functions. What evaluation design should the team add before promotion?
• Before promoting a multi-agent customer-support graph from staging to production in Microsoft Foundry, the quality team must score groundedness, tool-choice appropriateness, and tone on a fixed set of scenarios with human reviewers. What should you design first?
• Users report that a long-running multi-agent planning session “forgets” constraints agreed in the first ten minutes once the transcript grows large, even though no code bug resets state. Traces show oldest turns disappearing from the prompt assembly. Which context-window issue is this, and what is the best remediation direction?
• You need a continuous improvement loop for a multi-agent sales assistant. Human review capacity is limited, but you must detect quality regressions in grounded product claims weekly. Which approach aligns with AI-500 continuous improvement practices?
• A multi-agent research workflow fans out five specialist agents that each call the same Foundry model deployment. End-to-end duration is poor: some specialists idle while others hit HTTP 429s. What optimization strategy should you apply first?
• After enabling automatic conversation summarization between agent handoffs, auditors find that a critical contractual exclusion present in the original transcript is missing from later specialist contexts, replaced by a softer paraphrase that changes meaning. Which issue is occurring and what should you do?
• Incidents in production show incomplete explanations: Azure Monitor has some dependency logs, but engineers cannot reconstruct which orchestrator decision led which subagent to call which tool for a single user session. What observability design is required?
• A peer-to-peer multi-agent debate pattern occasionally enters long tool-call loops, driving token spend 20× above budget for a minority of sessions. You need controls that protect cost without killing legitimate multi-step work. What should you implement?
• Over three weeks, automated evaluations show a gradual drop in tool-selection accuracy after several prompt and model-version changes, though infrastructure uptime remains 99.9%. Which monitoring concern is this, and what should you add?
• Finance requires each business unit to pay for its multi-agent Foundry consumption. Today all agents share one deployment with no attribution, so chargebacks are impossible. What monitoring and management strategy should you design?
• You are building an evaluation suite for a multi-agent RAG helper that also calls MCP tools. Stakeholders want coverage beyond final answer text. Which evaluation design best matches the exam’s multi-agent validation scope?
• Platform engineers must alert when any multi-agent workflow exceeds a token budget or repeatedly fails a critical tool stage. Which Foundry-oriented tracing implementation details are essential?
• You want to stress-test a multi-agent assistant for rare but high-impact failures (prompt injection, wrong tool under urgency, contradictory memory) without waiting months for organic traffic. Which continuous improvement technique should you prioritize?