A financial regulator issues guidance stating that AI models used in credit underwriting must be 'explainable' to regulators on request. A bank uses a deep learning model for underwriting and argues that SHAP explanations satisfy this requirement. The regulator disagrees. What is the regulator's MOST likely concern with SHAP-based explanations?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — SHAP explanations are post-hoc approximations, not the model's actual decision logic. The deep model's internal computations stay opaque, so you can't independently verify how the decision was made.
Full Explanation
The regulator's concern is that SHAP explanations are post-hoc approximations that may not accurately represent the model's actual decision logic, and the model's internal mechanisms remain opaque to independent verification. SHAP approximates what the model 'would have done' with different features but does not provide a causal or mechanistic account of the decision. For regulatory examination, this opacity may be insufficient. The concern that SHAP values are computed too slowly for real-time explanations is practical but not the primary regulatory objection. The claim that SHAP values require the regulator to have access to the model's training data is wrong; training data access is not required for SHAP. The claim that SHAP explanations are only valid for tree-based models and cannot be applied to deep learning architectures is factually incorrect, because SHAP is applicable to neural networks.