During a pilot, complaint volume from one demographic segment is noticeably higher than from others, even though the model's overall accuracy metric looks healthy. What does this pattern suggest an administrator should do?

The correct response is to run an evaluation of outcomes broken out for that segment specifically, since a healthy aggregate metric can coexist with a real group-level problem. Uneven complaint volume by segment is a real-world signal of a possible group-level performance gap, and the way to confirm or rule it out is a disaggregated evaluation of that segment, not a re-reading of the aggregate figure.

The suggestion to dismiss the complaints as noise because the overall accuracy metric already accounts for every user is incorrect because the overall metric averages across everyone and specifically cannot show whether one segment is worse off. The suggestion to recalculate overall accuracy using a different rounding convention to see if the figure changes is incorrect because changing how the same aggregate number is rounded does not touch the underlying group-level question at all. The suggestion to wait for complaint volume to cross a fixed numeric threshold before investigating further is incorrect because waiting for an arbitrary threshold delays the investigation rather than informing it, and a real disparity can be small in volume yet still material.

For the exam, treat an uneven complaint pattern as a trigger for group-level evaluation, not as something an aggregate metric can explain away.