Domain 4: AI Audit & Assurance

ISACA AI Audit · 50 questions

• An AI auditor is designing a testing protocol for a loan underwriting model. Which approach BEST demonstrates that the model does not violate fair lending laws by using protected characteristics as proxies?
• An AI auditor is asked to evaluate the robustness of a recommendation system used in a streaming platform. Which test would BEST assess the model's vulnerability to adversarial manipulation?
• When conducting an AI system audit, which document provides the MOST essential baseline for assessing whether an AI system is operating within its intended scope?
• An organization wants to create an AI audit trail that satisfies regulatory requirements. Which data should be captured at minimum in the audit log for each AI decision?
• A CISO asks the AI risk team to assess the risks of using a third-party AI vendor for employee performance evaluation. Which due diligence element is MOST critical from an AI risk perspective?
• An AI model used for predicting hospital no-shows achieves 78% accuracy overall, but the AI risk team discovers it has a false negative rate of 35% for elderly patients. What action is MOST appropriate?
• An AI risk team is reviewing a third-party vendor's AI model. The vendor provides performance metrics but refuses to share training data, model architecture, or validation methodology. How should the team assess this situation?
• An AI audit program is being developed for a large bank. Which sequence of activities represents the CORRECT logical order for conducting an AI model audit?
• An AI model's validation report notes that the model performs well overall but fails on a specific edge case: when all optional fields in the input form are left blank. In an AI audit, how should this finding be classified?
• An AI auditor reviewing a sentiment analysis model used for customer complaint triage discovers that the model consistently misclassifies complaints written in informal language, slang, or non-standard spelling. This primarily impacts low-income and younger customers. What audit finding should be documented?
• A risk officer conducts a vendor assessment for a third-party AI fraud detection system. The vendor claims the model has been independently audited. Which follow-up question is MOST important?
• A healthcare organization is auditing an AI clinical decision support system. The auditors want to verify that the system was developed following responsible AI principles. Which documentation artifact provides the STRONGEST evidence that bias was assessed during development?
• An AI system for public benefits eligibility determination is suspected of denying benefits to eligible applicants at a higher rate than similar manual processes. Which AI audit procedure BEST assesses this concern?
• An organization is preparing its first AI audit program. The Chief Audit Executive asks what skills AI auditors need that traditional IT auditors may lack. Which capability is MOST uniquely required for AI auditing?
• A risk manager is reviewing an organization's AI incident response plan. Which element is MOST critical to include specifically for AI-related incidents that would not necessarily be required in a traditional cybersecurity incident response plan?
• An internal AI audit team at a bank is evaluating the scope of their AI model audit universe. A business unit argues that their rule-based credit scoring system, which was built by data scientists using statistical analysis, should be excluded because it is 'not really AI.' How should the audit team respond?
• An internal AI audit finds that a customer service AI model was promoted to production based solely on a business unit manager's sign-off, bypassing the formal validation process. The model governance policy requires independent validation for all customer-facing AI. Which audit finding classification is MOST appropriate?
• A risk committee is reviewing an AI model used to score job applicants for a technology company. The model was trained on profiles of historically successful employees, the majority of whom are male. The model's top scoring threshold accepts 28% of male applicants and only 14% of female applicants. Under the EEOC's uniform guidelines and the 4/5ths rule, what is the correct assessment?
• An auditor is reviewing the model validation report for a natural language processing model used in automated contract review. The validation report shows high performance on the vendor's internal test set but the deploying organization has not conducted its own independent testing. What is the MOST significant validation gap?
• During an internal AI audit, the team finds that a document summarization model deployed for legal discovery support occasionally omits key sentences from legal documents when the documents contain complex multi-clause sentences. What type of AI risk does this MOST represent, and what is the appropriate audit finding?
• An AI audit team is testing a credit decision model for stability. They create 1,000 synthetic applicant profiles that differ from real applicants by only one feature at a time and compare predicted scores. What technique is this?
• An AI auditor is reviewing evidence that an organization's AI model governance practices comply with their stated policy. Which audit evidence is MOST persuasive for demonstrating that model validation independence is maintained?
• An AI product team is developing a generative AI system that will produce patient education materials for a hospital. The team proposes releasing the system without formal validation because 'it's just generating text, not making medical decisions.' How should the AI risk manager respond?
• An AI system used by a government agency to assess eligibility for public housing produces systematically lower scores for families with non-English surnames. The agency's AI vendor claims the model does not use ethnicity or national origin as input features. Which investigation approach would BEST determine whether discrimination is occurring?
• Which element of the AI audit report is MOST important for ensuring that the organization's leadership can take informed corrective action following an AI audit?
• During an AI audit of a digital health platform, the auditor discovers that the AI model's outputs are stored in a non-auditable database with no query history, and logs are overwritten every 48 hours. What is the PRIMARY audit concern?
• An AI model audit team wants to verify that a model's published performance metrics accurately reflect its real-world performance. The model was built and evaluated by the same team. Which validation approach provides the STRONGEST evidence of real-world performance?
• An internal AI audit team is planning an audit of a natural language processing system used for contract review. Which approach BEST ensures audit coverage of model bias risk?
• During an AI system audit, the auditor wants to assess whether a training dataset used for a customer churn model is representative of the current customer population. Which audit procedure is MOST appropriate?
• An AI auditor is reviewing an organization's AI governance documentation. Which document type provides the STRONGEST evidence that an AI system's risks have been formally identified and accepted by accountable management?
• An audit of an AI-assisted underwriting system reveals that the model outputs a probability score, but underwriters must document their reasoning only when they override the model's recommendation. What control weakness does this reveal?
• When auditing a machine learning pipeline, which activity in the data preprocessing stage poses the HIGHEST risk of introducing systematic bias into the model?
• An AI auditor uses counterfactual testing during an audit of a hiring algorithm. Which of the following BEST describes what this testing technique reveals?
• An AI audit report identifies a finding that an organization lacks a documented AI model inventory. Under which audit reporting criterion should this finding be classified?
• An AI audit team is reviewing an organization's process for decommissioning retired AI models. Which risk is MOST critical to address in the decommissioning process?
• An AI auditor is assessing an organization's AI training data governance. The auditor finds that data scientists can modify training datasets without requiring documented approval. Which audit finding does this represent?
• When conducting an AI system audit, what is the MOST important reason to review the model's feature engineering documentation?
• An AI audit of a document classification system reveals the model achieves 95% accuracy overall but only 71% accuracy on documents from a specific business unit. What audit follow-up action is MOST appropriate?
• Which of the following BEST describes an adversarial test (red team exercise) in the context of AI system auditing?
• An AI auditor reviewing a model's post-deployment monitoring program finds that alerts are configured only for API error rates and latency. What significant monitoring gap does this reveal?
• Which principle BEST guides an auditor assessing whether an AI system's outputs are sufficiently explainable for the use case?
• During an AI model audit, the auditor requests access to the model's training data but is told it was deleted after training to reduce storage costs. What risk does this create for ongoing model governance?
• An AI audit plan is being developed for an organization that uses AI in three high-risk areas: credit underwriting, anti-money laundering transaction monitoring, and HR performance evaluation. Resources are insufficient to fully audit all three in one cycle. How should audit prioritization be determined?
• An AI audit team is evaluating the adequacy of a model's governance documentation. Which document would BEST provide evidence that a model was reviewed and approved for production use by an independent party?
• During an AI system audit, the auditor discovers that model outputs are reviewed by humans before being acted upon, but the human reviewers are processing 800 cases per day — far exceeding what one person could meaningfully review. What audit finding does this represent?
• An AI auditor is testing an AI model's robustness as part of an assurance engagement. Which test BEST evaluates the model's stability under minor input perturbations?
• When auditing an AI system's data pipeline, the auditor discovers that the final training dataset has a much higher proportion of positive class labels than the actual population. What risk does this create?
• An AI auditor is examining an organization's AI incident log. The log contains 47 entries over 12 months, all classified as 'minor.' The auditor interviews business units and discovers several significant AI failures that were not reported. What control failure does this reveal?
• Which of the following BEST describes the purpose of a 'bias bounty' program for a deployed AI system?
• A financial regulator asks a bank to explain why its AI credit model denied a specific customer's application. The model is a deep neural network with 50 layers. Which approach BEST satisfies the regulator's request while remaining technically honest?