Six weeks after launch, a multi-agent HR policy assistant still passes its original pre-release evaluation suite, but employees report that answers have become more verbose, less actionable, and occasionally cite retired policy versions after knowledge-base updates. Which monitoring strategy best closes this gap?
Select an answer to reveal the explanation.
Short Explanation
Frozen pre-release suites can't see what happens after knowledge-base edits and six weeks of prompt and traffic drift. You need to sample live conversations, measure behavior and groundedness, and alert when quality moves off a moving baseline. Leaving the knowledge base frozen forever blocks legitimate policy updates, and raising temperature makes verbosity and inconsistency worse.
Full Explanation
The best strategy is continuous production sampling with behavioral and groundedness metrics plus regression alerts that compare live quality to a moving baseline, not only the original offline suite. Evaluate/monitor guidance for multi-agent systems includes behavioral drift and quality regression detection against production traffic. Offline suites are necessary but insufficient when indexes, policies, and usage patterns change after launch. Continuous sampling with groundedness and behavioral metrics plus regression alerts detects silent degradation while the original suite still passes.
Freezing the knowledge base permanently so evaluation scores never change is incorrect because freezing knowledge indefinitely prevents required policy updates and is not a monitoring strategy. Celebrating the static suite pass rate as proof that production quality is unchanged is incorrect because a static suite that does not reflect new documents or live distributions can remain green while user-visible quality declines. Increasing temperature on all agents to restore creativity employees seem to want is incorrect because higher temperature increases randomness and often worsens verbosity and inconsistency; it does not monitor or correct grounding to current policies.