Which statement best summarises how fairness should be evaluated for a deployed AI system?
Select an answer to reveal the explanation.
Short Explanation
Disaggregate, and keep doing it. If a fairness risk exists, you measure outcomes separately for protected or relevant groups before launch and on an ongoing basis. That's the whole practice in one sentence.
Full Explanation
The statement that outcomes should be measured separately for protected or relevant groups wherever a fairness risk exists, before launch and on an ongoing basis, rather than inferred from an aggregate metric, is correct. Fairness is inherently a comparison between groups, so it requires measuring those groups separately, and it needs to be sustained over the system's life rather than treated as a single pre-launch gate.
The claim that a single accuracy or performance figure is sufficient is incorrect because an aggregate figure averages away exactly the differences a fairness evaluation is meant to detect.
The claim that fairness testing is only necessary once a regulator or customer raises a concern is incorrect because waiting for an external prompt turns evaluation into damage control instead of a proactive deployment control.
The claim that fairness testing can be limited to the group that makes up the largest share of the training data is incorrect because scoping to the largest group by definition excludes the smaller groups where a disparity is most likely to be found.
For the exam, be able to state both parts of this practice: evaluate by group, and keep evaluating after launch.