Untreated tickets are held as a control group to compare against AI-assisted handling during the pilot. What evaluation design is this?
Select an answer to reveal the explanation.
Short Explanation
Want to know if the bot helped? Keep a control stack of untreated tickets like a science fair. No baseline means you are guessing with confidence.
Full Explanation
Using comparative evaluation designs for municipal AI pilots is the correct evaluation design when untreated tickets are held as a control group to compare against AI-assisted handling during the pilot. That works because improvement claims need a counterfactual: without a non-AI baseline under comparable conditions, seasonal spikes, staffing changes, or process tweaks can be misread as model success. Holding untreated tickets as a control lets the city attribute differences in handle time, accuracy, or rework to AI assistance with far more credibility.
Claiming improvement without any non-AI baseline fails by concept: absolute numbers without comparison cannot prove the AI caused the change. Comparing only to the vendor's marketing case study fails by concept—external marketing is not a matched control in this city's queues, content, and workforce. Changing every ticket workflow mid-pilot so nothing is comparable fails by concept: simultaneous uncontrolled changes destroy internal validity of the evaluation.
Exam caveat: controls must be ethically and operationally sound (fair assignment, no resident harm from withheld help when policy forbids it) and documented. Keep municipal ticket / pilot evaluation framing. Operational check: define treated vs untreated assignment rules; freeze confounding workflow changes during the comparison window; pre-register metrics; analyze differences with the control; and archive the design so promotion decisions cite evidence, not anecdote.