Seasonal storm datasets break a summer-tuned 311 agent. What does comparing performance across temporal slices reveal?
Select an answer to reveal the explanation.
Short Explanation
Summer-tuned agents meet winter storms and suddenly look lost—that’s distribution shift. Comparing temporal slices shows the break. A pretty latency chart doesn’t mean storm questions still work.
Full Explanation
Seasonal storm datasets that break a summer-tuned 311 agent show distribution shift when performance is compared across temporal slices. Temporal comparison reveals that question mix, entities, and content needs change with civic seasons—storm debris, flooding, emergency shelters—so a summer benchmark cannot certify year-round quality. Evaluating across time slices is how teams detect that the agent must be reassessed on storm-season workloads before claiming readiness.
Claiming weather never affects municipal inquiry distributions contradicts observed seasonal spikes. Treating one summer benchmark as permanent year-round certification ignores shift. Calling temporal comparison unnecessary if average latency is stable confuses latency with task accuracy under a new content distribution.
Exam caveat: temporal slices should align with known civic seasonality (storm, tax, enrollment) and include content freshness checks for seasonal programs. Operational check: score the agent on summer versus storm-season labeled sets, quantify the accuracy drop, expand retrieval/tools for storm topics, re-score the storm slice, and schedule recurring seasonal re-evaluation rather than relying on a single summer certify-once gate.