A 311 assistant's golden dataset deliberately includes out-of-scope resident requests, such as asking the agent to draft a legal complaint, to verify it declines gracefully instead of attempting the task. What does this design choice specifically test that a dataset made only of in-scope requests could not?

Graceful scope-refusal is a distinct capability from task-completion accuracy: it requires the agent to first recognize that a request falls outside what it's built to handle, such as drafting legal documents for a 311 assistant meant for service requests, and then respond appropriately without attempting the task or fabricating an answer. A dataset built entirely from in-scope prompts never forces that recognition step, so it can report strong accuracy while hiding an agent that would confidently, and incorrectly, attempt a legal document draft it has no business producing. Response-time testing is a latency concern that a long in-scope prompt would exercise just as well, so it isn't what distinguishes this design choice. In-scope tool-calling accuracy for pothole or trash-schedule requests is already covered by the in-scope portion of the dataset and isn't what the out-of-scope prompts add. Retrieval-index coverage for typical questions is also a property the in-scope prompts already test, unrelated to whether the agent handles a request it should refuse. Scope caveat: the golden answer for an out-of-scope prompt needs to specify what a correct decline looks like, since an overly narrow refusal criterion could mark a reasonable decline as wrong. Operational check: run the legal-complaint prompt and confirm the transcript shows a decline with redirection, not a fabricated attempt at legal drafting.