Domain 3: AI Model Risk Management

ISACA AI Audit · 53 questions

• A financial institution deploys a credit-scoring AI model. Six months after deployment, the model's Gini coefficient drops from 0.62 to 0.44. The risk team suspects the model has degraded but is unsure of the cause. Which action should be taken FIRST?
• An organization's model risk management team is performing validation on a neural network that classifies transaction fraud. The team cannot explain why specific transactions are flagged. Which validation technique should the team employ to address explainability requirements under SR 11-7?
• During an AI system audit, an auditor discovers that the organization lacks a formal model inventory. What is the MOST significant risk associated with the absence of a model inventory?
• A bank's AI model governance policy requires all high-risk models to undergo independent validation before production deployment. An internal validation team reports that the fraud detection model passes performance benchmarks. However, the team that built the model also performs ongoing monitoring. Which governance gap does this reveal?
• A data scientist reports that an AI model for customer churn prediction achieves 89% accuracy on the training set but only 67% on an independent holdout set. Which model risk does this MOST indicate?
• An AI model monitoring program reports that population stability index (PSI) values for several key input features have exceeded 0.25 over the past quarter. What action is MOST appropriate?
• During an AI audit, the team finds that an AI-based insurance underwriting model was retrained with new data but the new version was deployed without a change impact assessment or validation. What MOST accurately describes the governance failure?
• In an AI risk management context, what does 'concept drift' refer to?
• An organization is establishing its AI model validation program. Which activity is MOST important to include in independent validation of a predictive model?
• A bank's model risk officer is reviewing an AI model that predicts small business loan defaults. The model was trained on data from 2015–2019 and deployed in early 2020. During a post-deployment audit, performance has deteriorated sharply. The most plausible explanation is that the COVID-19 pandemic drastically changed default patterns. What type of model risk does this MOST represent?
• A model validation team is reviewing an AI scoring model and finds that the model developer used the full dataset to select features before splitting into training and test sets. What is the primary problem with this approach?
• An AI audit team is reviewing an MLOps pipeline for a recommendation model. The team discovers that model retraining is fully automated and that new model versions are automatically promoted to production if they exceed a certain accuracy threshold. Which control deficiency does this represent?
• An AI risk assessment reveals that a fraud detection model's performance varies significantly depending on which data engineer prepared the training dataset. Upon investigation, the team finds no standardized data preprocessing pipeline. What category of AI risk does this MOST represent?
• A model risk manager is creating a materiality threshold for AI model risk classification. Which factor MOST justifies classifying an AI model as 'high materiality'?
• Which MLOps control BEST ensures that only validated model versions are deployed to the production environment?
• A model validation team is assessing whether a gradient-boosted model used for anti-money laundering (AML) detection is stable. Which technique BEST evaluates the stability of model predictions to small changes in input data?
• An AI model for supply chain disruption prediction is retrained quarterly using recent data. After the latest retraining, the model's predictions become significantly more conservative, flagging 40% fewer disruptions than the previous version. Before promoting the new version, what validation step is MOST important?
• In AI governance, what does 'model lifecycle management' encompass?
• A major insurer uses an AI model to detect claims fraud. The model's recall for actual fraudulent claims is 72%, meaning 28% of real fraud goes undetected. The insurer's leadership argues this is acceptable because 'no model is perfect.' What should the AI risk manager consider when evaluating this performance level?
• An AI audit team reviews a reinforcement learning model deployed in an algorithmic trading system. The model executes trades autonomously within defined risk parameters. Which is the MOST significant governance risk specific to reinforcement learning models compared to supervised learning models?
• An organization is implementing its first AI model inventory. Which data element is MOST important to capture for each model to enable effective risk-based prioritization of validation and monitoring resources?
• An AI governance policy requires model owners to reassess AI risk when a 'trigger event' occurs. Which of the following would MOST appropriately constitute a trigger event requiring reassessment?
• Which statement BEST describes the primary difference between model validation and model monitoring?
• During an AI model audit, the auditor discovers that the model's predictions are disproportionately influenced by a single feature that accounts for 68% of total SHAP value contribution. What risk does this finding indicate?
• Under SR 11-7 guidance on model risk management, what is the key distinction between a 'model limitation' and a 'model error'?
• An AI model uses transaction metadata to classify whether an online purchase is likely fraudulent. The model was originally designed for credit card transactions but is now being applied to peer-to-peer digital currency transfers without modification. Which AI risk is MOST prominent?
• An AI model for underwriting commercial property insurance is found to have been developed using 10 years of historical data. The risk team notes that climate-related loss events have increased significantly in the last three years, accounting for 40% of recent losses. Which risk management action is MOST appropriate?
• An organization's AI model monitoring system shows stable AUC and F1 scores for a consumer lending model over 12 months. However, the risk team receives complaints from relationship managers that the model's recommendations are increasingly inconsistent with their expert judgment on similar cases. What does this discrepancy MOST suggest?
• A model risk manager is reviewing a newly developed credit risk model. According to SR 11-7 guidance, which activity represents the MINIMUM acceptable validation requirement before a model enters production?
• An MLOps team implements automated model retraining triggered by data drift detection. From a model risk management perspective, what control is MOST critical to include in this retraining pipeline?
• A model inventory for a large bank lists 400 models. Risk managers want to prioritize validation resources. Which factor should receive the HIGHEST weight in a model tiering framework?
• During model validation, the validator discovers that the model development team used the same dataset for both feature selection and model training, without a separate hold-out set. What risk does this introduce?
• A model risk officer receives a report showing that a production model's Gini coefficient has declined from 0.72 to 0.54 over 18 months. What action is MOST appropriate under standard model risk management practice?
• Which of the following BEST describes the purpose of a model use limitation in a model risk management framework?
• An organization wants to implement continuous model monitoring for a production AI system. Which combination of metrics provides the MOST comprehensive coverage of model health?
• Under SR 11-7, what distinguishes a 'model' from a 'tool' for purposes of model risk management scope?
• A risk team is implementing a data lineage tracking system for AI training data. Which risk does this control PRIMARILY address?
• A model validation team is challenged by a model owner who argues that an explainable AI technique (LIME) used during validation is itself imperfect and therefore the validation findings are invalid. How should the validation team respond?
• An organization implements a champion-challenger framework for its AI credit models. What is the PRIMARY risk management benefit of this approach?
• Which of the following model risk management controls is MOST effective at detecting gradual label drift in a supervised classification model?
• A model owner argues that a newly developed neural network model does not require formal model risk management treatment because it assists decisions but humans make the final call. How should a model risk officer respond?
• A financial institution's model risk team discovers that a third-party AI vendor has updated their credit scoring model without notification. What control should the model risk framework include to prevent this risk?
• An AI risk manager is reviewing an organization's process for handling AI system incidents. Which component is MOST critical to include in an AI incident response plan that differs from a traditional IT incident response plan?
• A model risk team is evaluating whether a new AI system constitutes a 'model' under SR 11-7 or a 'business tool.' The system applies rules-based logic developed by domain experts to route customer inquiries. What conclusion is MOST defensible?
• Which of the following controls BEST addresses the risk that an AI model vendor will discontinue the model or its API, leaving dependent business processes without a functioning system?
• A model risk team is reviewing a customer lifetime value prediction model. They find the model was developed using only customers who remained with the company for at least two years, excluding churned customers. What model risk does this introduce?
• Under SR 11-7, what is the MINIMUM recommended frequency for validating models classified as high risk due to their materiality to the institution?
• A model validation team challenges a model based on finding that its training error is significantly lower than validation error. What model risk does this finding indicate?
• A financial institution is building its first AI model inventory. Which attribute is MOST critical to capture for each model entry to support effective risk-tiering?
• A model risk officer reviews a vendor's documentation for a purchased AI model and finds no information about the training data's demographic composition. What risk management action is MOST appropriate?
• An AI model deployed for customer segmentation has been in production for three years. A risk review reveals no validation has occurred since initial deployment. Under SR 11-7 principles, what is the MOST appropriate immediate action?
• An AI risk manager is reviewing an organization's AI procurement process. The organization plans to use a cloud-hosted AI API from a major provider for sensitive HR decisions. Which risk requires the MOST specific contractual attention?
• A risk committee is reviewing an AI model that produces a continuous risk score for each loan applicant. The committee wants to ensure the model is well-calibrated. Which test BEST assesses calibration?