A five-agent orchestrated pipeline completes successfully (no exceptions thrown), but the final output is incorrect. No error logs were generated. Which evaluation signal would most likely detect this failure earliest?
Select an answer to reveal the explanation.
Short Explanation and Infographic
A conveyor belt with no quality checkpoints can ship defective products even though the belt itself never jammed. On the exam: silent correctness failures require explicit success criteria checked at each stage — structural schema checks and timing monitors miss semantic failures.
Full explanation below image
Full Explanation
Silent failures — where the pipeline completes without errors but produces incorrect output — are the hardest failure mode to detect. The key insight is that 'no exception' is not the same as 'correct output.' The correct approach (option C) is to define explicit, measurable success criteria for each intermediate agent output — for example, 'the refactored function must pass its test suite,' or 'the synthesized summary must include at least three cited sources' — and run automated checks after each stage. This catches semantic failures, not just structural ones, and localizes the failure to the stage where it occurred.
Option B (schema validation) is valuable but insufficient. Schema validation checks that data is in the right format, not that the data is correct. An agent can return a perfectly valid JSON object that contains semantically wrong information — schema checks will pass while the output is still incorrect.
Option A (execution time monitoring) is even weaker. Timing anomalies can hint at problems but are highly indirect signals. An agent can produce wrong output in the same time as correct output. Timing baselines also vary widely based on input complexity.
Option D (verbose logging) helps with post-hoc diagnosis but cannot detect a silent correctness failure in real time. If no exception was thrown, verbose logs will show the pipeline running normally — there is nothing in the logs that indicates the output is semantically incorrect. Logging is a diagnostic tool, not a detection mechanism for silent failures.