Eval, Testing, and Debugging

CCDV-F · 5 questions

• A 311 on-call engineer sees repeated tool-schema validation errors before Claude replies. How should the fault be classified?
• A permitting bot returns fluent but wrong zone codes even though traces show correct tool data returned. What failure type fits?
• A transit agent hits Claude API rate limits during a rush-hour outage FAQ surge. Which recovery strategy fits first?
• A court assistant debug session must find why Claude never saw a filing-status tool reply. What analysis step fits?
• A public-health eval harness flags rising hallucination rates after a prompt edit. What recovery action fits?