A city staggers a new permit web form to some applicants while others still use the legacy form, then compares completion rates. What comparative-evaluation caution applies?
Select an answer to reveal the explanation.
Short Explanation
A staggered rollout is a chance to compare, not a free pass to declare victory. Watch for confounders—different neighborhoods, seasons, or traffic mixes—before crowning the new form. Careful comparison beats hasty causation.
Full Explanation
When a city staggers a new permit web form to some applicants while others still use the legacy form and then compares completion rates, the comparative-evaluation caution is to compare outcomes across treatments carefully while watching for confounders. Staggered rollouts can inform UX and process decisions if outcome definitions are sound and differences in applicant mix, seasonality, channel, device, or marketing are considered before citywide mandates. Treating any completion-rate difference as proof the new form is causally superior with no further checks fails by overclaiming from an imperfect natural experiment. Skipping comparison entirely because staggered rollouts cannot inform UX decisions fails by wasting a useful learning design. Merging both forms’ events into one unlabeled completion rate before comparing fails by destroying the treatment contrast needed for evaluation. Exam caveat: awareness-level A/B thinking does not require a perfect randomized trial for every UX change, but it does require honest uncertainty language and attention to confounders before scaling. Operational check: define completion consistently, segment comparison groups by known confounders, report effect sizes with caveats, avoid merging treatments into one unlabeled rate, and only scale the new form citywide after review of bias risks and operational side effects.