A benefits chatbot is evaluated for different error rates across language groups and ZIP codes before wider release. What ethical practice is this?
Select an answer to reveal the explanation.
Short Explanation
You’re checking for biased outcomes by comparing error rates across language groups and ZIP codes before the benefits chatbot reaches more people. Citywide averages can hide who gets the bad answers, like checking every neighborhood’s water quality rather than relying only on the reservoir average.
Full Explanation
Testing civic AI for biased outcomes across communities is the ethical practice when a benefits chatbot is evaluated for different error rates across language groups and ZIP codes before wider release. Citywide averages can hide concentrated harm: a chatbot that performs well overall may still misroute or misinform language minorities or specific neighborhoods at higher rates. Disaggregated evaluation before scale-up helps the city find and remediate disparate error patterns—content gaps, retrieval failures, or UI barriers—before they become production equity problems in benefits access. Assuming equal accuracy because the model is large fails because size does not guarantee equity or measure subgroup performance on municipal content. Checking only average citywide accuracy and stopping there fails because averages mask subgroup harm and can create false confidence about release readiness. Excluding language minorities from the test sample to simplify results fails both conceptually and ethically: it removes groups most at risk and manufactures a biased “pass” instead of measuring real service quality. Exam caveat: disparity findings require remediation plans, such as content improvements, translation, and escalation paths—not just a memo noting disparities. Practical check: define language and geography slices, sample enough cases in each, compare error and escalation rates, block wider release when gaps exceed thresholds, and retest after content or model changes before expanding.