A public-health model audit measures overall accuracy but does not sample across demographic strata. What testing gap should the auditor cite?

Citing missing stratified sampling needed for fairness-related AI audit tests works because aggregate accuracy can conceal underperformance for specific demographic or geographic groups served by public-health models. Fairness conclusions require intentional sampling or evaluation across relevant strata, not only pooled metrics. Without stratification, a clinic model may look strong citywide while harming a neighborhood or population subgroup.

Missing robustness testing for temporal distribution shift effects fails as a fairness testing approach because temporal drift is unrelated to whether model accuracy varies across demographic or geographic groups. Incomplete privacy leakage assessment of training data extracts fails as a fairness testing approach because data-privacy review is unrelated to whether model accuracy varies across demographic or geographic groups. A lack of inter-rater reliability testing among the clinical labelers is a real data-quality concern, but it does not address whether the model's accuracy actually varies across demographic or geographic strata, which is the specific fairness testing gap the audit is asking about.

Exam caveat: stratum definitions must be justified, privacy-preserving, and relevant to the model’s use; arbitrary demographic slicing without hypothesis or policy basis can be both ineffective and sensitive. Operational check: define fairness strata tied to the public-health use case, ensure lawful access to needed attributes or proxies, sample or evaluate within strata, and compare error/impact metrics—not only overall accuracy—before concluding the model is equitable.