A courts-clerk scheduling agent handles multi-turn conversations with self-represented litigants. Its evaluation suite scores not just each turn's individual accuracy but also whether the agent maintains correct case context across the whole conversation. What failure would per-turn accuracy scoring alone miss that this added check catches?
Select an answer to reveal the explanation.
Short Explanation
Judging each turn alone is like grading someone's answers to a set of trivia questions without noticing they've quietly started answering about a different topic. Context-continuity checks are what catch the agent that sounds fine turn by turn but has actually lost the thread of which case it's even discussing.
Full Explanation
Multi-turn conversational agents carry state across turns, and per-turn accuracy scoring evaluates each reply in isolation against what a good response to that single message would look like, which can miss a failure that only shows up in relation to earlier turns — such as the agent replying coherently but about the wrong case because it silently lost track of context established several turns earlier. Context-continuity evaluation specifically checks whether the case identity, filing details, or other state carried from earlier turns remains correct later in the conversation, which is a property of the conversation as a whole rather than of any single reply. A slow individual response is a latency issue, measurable per turn and unrelated to whether context persisted. A misspelled case number in one isolated turn is a single-turn accuracy defect, catchable without any cross-turn comparison. A retrieval miss on one specific turn is also a per-turn defect, detectable by scoring that turn alone against its own golden answer. Scope caveat: context-continuity scoring needs conversation-level golden transcripts, not just isolated prompt-response pairs, or there is nothing to check drift against. Operational check: run a multi-turn golden conversation and confirm a later turn still references the same case identifier introduced in an earlier turn.