A city evaluates its benefits-scoring foundation model for differences in outcomes across demographic groups before wider rollout, beyond just checking its overall accuracy score. Why is this bias-detection step necessary in addition to overall accuracy?
Select an answer to reveal the explanation.
Short Explanation
Think of a citywide average commute time — it can look perfectly reasonable even if one neighborhood's residents are stuck in traffic every single day, because the average just smooths that over. Overall accuracy works the same way: a model can score well in aggregate while still treating specific demographic groups very differently underneath that average. That's exactly why the city has to look at subgroup outcomes separately instead of trusting one overall number.
Full Explanation
Overall accuracy is an aggregate measure that can mask meaningful differences in how a model performs across specific demographic subgroups; a benefits-scoring model could achieve strong overall accuracy while systematically under-approving or mis-scoring one group more than another, and that disparity would not be visible in a single aggregate number. This is exactly why bias-detection evaluation, which examines outcomes broken out by subgroup, is a necessary complement to overall accuracy, particularly for a high-stakes benefits-scoring system where unequal treatment has real consequences for residents. The option ruling bias detection out of scope for foundation-model systems is incorrect, since foundation models can encode and reproduce demographic disparities from their training data or fine-tuning just as traditional ML models can, making subgroup evaluation equally relevant. The option claiming overall accuracy already guarantees demographic fairness misunderstands what an aggregate metric measures; accuracy summarizes performance across the whole population and says nothing on its own about how errors are distributed among subgroups. The option claiming subgroup evaluation should replace accuracy entirely overcorrects; the two serve different purposes, and dropping aggregate accuracy would lose the overall performance signal that bias detection is meant to complement, not substitute for. Scope note: bias metrics should be selected to match the specific fairness definition relevant to the use case, since different fairness metrics can conflict with one another. Operational check: compare approval or scoring rates across demographic subgroups against expected baselines before approving the model for wider rollout.