An AI system for public benefits eligibility determination is suspected of denying benefits to eligible applicants at a higher rate than similar manual processes. Which AI audit procedure BEST assesses this concern?
Select an answer to reveal the explanation.
Short Explanation
You need to compare AI denial rates to manual review denial rates across demographic subgroups, then review a sample of AI denials case by case. That directly tests whether the AI is denying eligible applicants at a higher rate and whether individual denials were accurate.
Full Explanation
Comparing AI denial rates to manual review denial rates across demographic subgroups directly tests the hypothesis that the AI is performing differently from the established process. Case-by-case review of a sample of AI denials provides evidence of whether individual denials were correct or erroneous. Reviewing the AI system's code to check for programming errors in the eligibility logic can find implementation bugs but won't reveal emergent discriminatory patterns in model outputs. Testing the AI system's response time and throughput capacity under peak load assesses operational capacity, not decision accuracy. Confirming that the AI system's training data was collected and stored in compliance with applicable privacy laws is a separate concern from decision accuracy.