Operations reports that a multi-agent workflow’s quality slowly worsened over two weeks without a code deploy. Tool schemas are unchanged. You need a monitoring strategy that catches this class of failure and supports response. What should you design?
Select an answer to reveal the explanation.
Short Explanation
You need to monitor for behavioral drift and quality regression with baselines on key eval metrics, correlate with traces, alert on sustained degradation, and define automated or runbook remediation such as pinning model version, disabling a flaky tool path, or rolling back prompt config. Quality can drift from model updates, data shift, or prompt-config changes even without app code deploys. Region-down-only alerts miss silent quality death, and paging on every success is noise.
Full Explanation
Monitor for behavioral drift and quality regression with baselines on key eval metrics, correlate with traces, alert on sustained degradation, and define automated or runbook remediation (pin model version, disable a flaky tool path, or roll back prompt config). Multi-agent reliability monitoring includes health, workflow failures, trace correlation, drift detection, quality regression, and remediation. Slow quality loss without code deploys is classic behavioral drift; baselines, alerts, correlated traces, and predefined remediation actions are appropriate. Region-down-only alerting is wrong because platform outage alerts do not detect gradual quality regression. Assuming quality cannot drift without a code change, so monitors should be disabled, is wrong because model, data, and config changes can shift behavior without application code commits. Paging on-call for every successful agent turn is wrong because alerting on successes creates noise and hides real incidents. Exam tip: drift detection plus baselines plus remediation runbooks, not only infra uptime.