311 plans a citywide chatbot expansion. What pilot design practice should precede that expansion?

The correct answer is running a time-boxed pilot with success metrics and explicit kill criteria before citywide 311 chatbot expansion. Pilot design discipline forces the city to define what “good” means (deflection quality, resident satisfaction, escalation accuracy, equity across languages and channels), when the pilot ends, and under what measured failures expansion stops. Kill criteria protect the public from locking in a harmful or ineffective assistant under political momentum, while success metrics make the go/no-go decision evidence-based rather than anecdotal.

Expanding citywide on day one with no evaluation window fails by concept because it skips evidence gathering and multiplies blast radius across all residents at once. Declaring success based only on executive enthusiasm fails because enthusiasm is not a performance metric and cannot detect equity or accuracy failures. Hiding failure metrics from stakeholders until after full rollout fails because it blocks informed oversight and prevents a legitimate stop decision.

Exam caveat: pilot questions on CRAGE almost always expect defined metrics plus exit or kill criteria, not a vanity go-live. Operational check: write a pilot charter with dates, KPIs, equity slices, and kill thresholds; brief stakeholders on failure metrics in advance; retain pilot logs; and require a documented decision gate before citywide expansion.