A risk committee is reviewing an AI model that produces a continuous risk score for each loan applicant. The committee wants to ensure the model is well-calibrated. Which test BEST assesses calibration?
Select an answer to reveal the explanation.
Short Explanation
You need to compare predicted probabilities to actual outcomes. The calibration plot and Brier score show whether a model predicting 20% default actually defaults 20% of the time. ROC-AUC checks ranking, not calibration.
Full Explanation
The calibration plot and Brier score approach is correct because calibration assessment requires comparing predicted probabilities to actual observed outcomes. A calibration plot, also called a reliability diagram, and the Brier score directly measure whether a model that predicts 20% default probability actually defaults at 20% in practice. The ROC-AUC comparison against a logistic regression baseline measures discriminatory power, not calibration. The permutation importance methodology measures feature contributions. The holdout dataset from a different geographic region tests geographic generalization, not calibration.