AI Pilot Execution and Scaled Deployment

CAIPM · 20 questions

• The permits office runs a time-boxed FAQ chatbot pilot with written success metrics and exit criteria. What pilot practice is this?
• The chatbot pilot answers only two permit types before any plan to cover the full taxonomy. What pilot practice is this?
• Pilot KPIs include deflection rate, sampled answer accuracy, and clerk satisfaction instead of only login counts. What measurement practice is this?
• Untreated tickets are held as a control group to compare against AI-assisted handling during the pilot. What evaluation design is this?
• Promotion from pilot requires passing an accuracy threshold, privacy sign-off, and a defined support model. What governance gate is this?
• A pilot misses its metrics; leadership documents lessons and stops it instead of quietly widening scope. What pilot discipline is this?
• Resident notices explain that the permitting assistant is in pilot and how to reach a human clerk. What pilot communication practice is this?
• The 311 assistant rolls out to one district's queues first, then citywide only after readiness checks pass. What deployment approach is this?
• Before production traffic, the team confirms runbooks, on-call coverage, and a tested rollback for the chatbot. What readiness practice is this?
• Feature flags let operations disable GenAI drafting quickly if grounding quality drops in production. What deployment control is this?
• A city AI program manager will not start the next 311-assistant rollout wave until clerks finish refresher training and the SOP packet is updated. What readiness practice is this?
• Phase two of a parcel-lookup AI assistant is blocked until parcel-ID mismatches in the land system are cleaned up. What rollout discipline is the program applying?
• After a permitting chatbot go-live, the city schedules two weeks of extra clerk coverage to catch AI mistakes in real time. What deployment practice is this?
• Before expanding a GenAI drafting tool from a pilot bureau to citywide traffic, the program runs a live drill that disables the feature and restores the prior workflow. Why?
• Departments are racing to stand up separate AI assistants with different grounding rules and no shared evaluation. What expansion risk should the AI program manager flag?
• After the council updates permit fees, the chatbot keeps quoting last year’s amounts until someone notices. What scale-risk control was missing?
• A successful 311 bot deflects routine questions but drives a surge of edge-case tickets that clerks must research. What expansion risk should leadership plan for?
• A citywide assistant launches in English only while large resident communities primarily use other languages. What scale risk must the program mitigate?
• During a storm-related 311 surge, the AI assistant starts failing because the vendor hits rate limits. What capacity practice should have been in the scale plan?
• A FAQ chatbot is expanding to give preliminary eligibility guidance for housing assistance. What governance action should accompany that scale step?