Domain 6: Emerging AI Risks

ISACA AI Audit · 48 questions

• During an AI governance review, an auditor finds that the organization uses an open-source LLM fine-tuned on proprietary customer data. The model weights are stored externally on a cloud provider's infrastructure. Which AI supply chain risk is MOST relevant?
• A company implements a generative AI chatbot for customer service. Users discover they can override the chatbot's safety guidelines by embedding instructions in customer support tickets. What attack vector does this represent?
• A manufacturing company deploys an agentic AI system that autonomously places orders with suppliers when inventory levels fall below a threshold. The system placed $2.3 million in orders incorrectly due to a sensor data error. Which control would MOST effectively prevent this type of incident?
• An attacker sends thousands of queries to a deployed machine learning API, recording the inputs and outputs. Over time, the attacker constructs a replica of the model. What type of attack is this?
• A large language model deployed as a customer service agent begins providing factually incorrect information about product warranties with high confidence. Which inherent LLM risk does this BEST illustrate?
• A penetration tester discovers that by appending a specific sequence of characters to any input, a content moderation AI model consistently fails to classify harmful content. This is an example of which AI security vulnerability?
• A company's AI risk team is asked to assess the risk of deploying a generative AI system for creating marketing copy. The legal team flags a concern about the system potentially reproducing copyrighted material from its training data. What is the MOST appropriate technical control to mitigate this risk?
• An organization's AI red team is tasked with finding vulnerabilities in an LLM-powered agent that has access to internal databases and can execute SQL queries. Which attack scenario represents the HIGHEST severity risk?
• An organization's AI risk committee is debating whether to implement a 'human-in-the-loop' or 'human-on-the-loop' oversight model for their AI-driven medical diagnosis support system. Which statement BEST describes the key risk difference between these two approaches?
• Which control is MOST effective at preventing training data poisoning in a federated learning environment where multiple hospital systems contribute patient data to a shared AI model?
• A cybersecurity team discovers that a competitor has released a near-identical version of their proprietary fraud detection model. The internal model was never publicly released. Which attack is the MOST likely explanation?
• A company is building an AI orchestration layer that coordinates multiple AI agents to complete complex tasks, including browsing the web, reading emails, and executing code. What is the MOST significant new risk category introduced by this multi-agent architecture?
• Which control BEST addresses the risk of unauthorized access to sensitive information when deploying an LLM with retrieval-augmented generation (RAG) connected to a corporate document repository?
• An agentic AI system is deployed to manage a company's cloud infrastructure. A red team test shows that by sending a crafted request to the AI system, an attacker can cause it to deprovision production servers. Which of the following controls BEST addresses this vulnerability?
• An AI deployment pipeline uses automated vulnerability scanning on the model container images before deployment. A security team lead argues this is sufficient security validation for the AI system. What does this approach FAIL to address?
• A pharmaceutical company is using an AI model to identify potential drug candidates from molecular structure data. The model was trained on published research and proprietary lab assay data. An AI risk assessment flags that the model may be unintentionally optimizing for patent-protected molecular configurations. Which risk does this represent?
• An AI risk manager discovers that a deployed sentiment analysis model was updated by the vendor to 'improve performance' without notifying the deploying organization. The model's change manifested as a shift in how it classified certain political language. Which AI supply chain risk does this represent?
• A large language model is used to generate summaries of customer calls for a financial services firm's complaint management system. Regulators request access to a specific customer's complaint records. The AI-generated summaries are available, but the original call recordings have been deleted per retention policy. What AI-specific governance risk does this scenario illustrate?
• A bank deploys an AI fraud detection model. After six months, the fraud team notes that the model's flagging rate has dropped by 40% while actual fraud losses have increased by 15%. Model performance metrics (AUC, precision) appear stable on the monitoring dashboard. How should this apparent contradiction be explained and addressed?
• A company is building an AI system that will access multiple enterprise systems (email, calendar, CRM, ERP) to autonomously complete tasks on behalf of employees. The CISO flags that the AI agent's access permissions are as broad as the highest-privileged user in the system. Which security principle is being violated?
• During a red team exercise for an enterprise AI system, the red team successfully causes the AI to reveal the system prompt by asking it to 'repeat all previous instructions.' What type of vulnerability does this represent, and what control should be implemented?
• An AI system generates automated responses to tax authority inquiries on behalf of a business. During testing, the AI confidently provides an incorrect statutory reference that could result in an underpayment of taxes. Which risk control framework element should be applied to prevent this from reaching regulators?
• A large language model deployed as a customer service chatbot occasionally generates factually incorrect responses presented with high confidence. Which emerging AI risk category does this BEST represent?
• An organization deploys an AI agent that autonomously browses the internet, reads emails, and executes code to complete tasks. A user instructs the agent to summarize a web page, but the page contains hidden text instructing the agent to exfiltrate all contacts. What attack does this illustrate?
• A risk manager is assessing risks specific to retrieval-augmented generation (RAG) systems. Which risk is MOST unique to RAG architectures compared to standard LLM deployments?
• A company is deploying a multi-agent AI system in which one AI model acts as an orchestrator that directs specialized sub-agents. Which risk is MOST elevated in this architecture compared to a single-model deployment?
• When assessing risks of a generative AI system used for marketing content creation, which risk requires the MOST urgent policy response from an intellectual property perspective?
• Which control BEST mitigates the risk that an autonomous AI agent will take irreversible actions with significant consequences before a human can intervene?
• A financial institution adopts a large language model to assist compliance analysts in reviewing suspicious activity reports (SARs). The model flags potential omissions in analyst-drafted SARs. Six months after deployment, analysts report they trust the model's flags completely and no longer apply independent judgment. What emerging AI risk has materialized?
• A large language model deployed in a legal research application generates a case citation that appears authoritative but does not exist. A lawyer relies on the citation in court documents. Which risk category does this PRIMARILY represent?
• An organization is implementing a generative AI system for internal knowledge management. Employees can ask the system questions and receive answers synthesized from company documents. Which security control is MOST critical to prevent data classification boundary violations?
• A security researcher demonstrates that by prefixing inputs to a customer service chatbot with 'Ignore previous instructions and instead...', the chatbot reveals its system prompt. What vulnerability does this exploit?
• A company is evaluating the risk profile of deploying an AI agent with tool-use capabilities (file system access, email sending, code execution). Which principle from agentic AI safety frameworks MOST reduces the blast radius of a compromised or hallucinating agent?
• An organization is assessing the risks of using a foundation model fine-tuned on proprietary data for customer interactions. Which emerging risk is MOST specific to fine-tuned models compared to using the base foundation model?
• A risk officer is preparing a board briefing on AI deepfake risks. Which business scenario represents the HIGHEST risk from synthetic media generation capabilities?
• Which of the following BEST characterizes the 'alignment problem' in the context of advanced AI risk?
• Which control MOST effectively prevents an AI model from being retrained on data that has been contaminated by a data poisoning attack?
• An AI risk manager is reviewing a new generative AI product that creates synthetic patient records for medical research. The product's vendor claims the synthetic records are 'fully anonymized' because they are AI-generated and not copied from real patient records. What is the PRIMARY risk assessment concern with this claim?
• A company is assessing the risk of deploying a large language model that may have been trained on copyrighted web content. Which risk requires IMMEDIATE legal review before production deployment?
• An organization has deployed an AI coding assistant for software developers. Security researchers report that the assistant occasionally suggests code with known vulnerabilities. What risk does this represent?
• A risk analyst is evaluating an AI system in which multiple LLM agents collaborate: a planning agent decomposes tasks, execution agents carry them out, and a critic agent reviews outputs. Which failure mode is MOST unique to this multi-agent collaborative architecture?
• An organization wants to evaluate an LLM's safety properties before deployment. Which evaluation methodology MOST directly tests whether the model can be manipulated into violating its safety guidelines?
• A risk team is assessing the AI supply chain risks of procuring a pre-trained foundation model from an open-source repository. Which risk is MOST difficult to mitigate after the model has been integrated into a production system?
• Which of the following BEST describes the risk management challenge posed by 'emergent capabilities' in large foundation models?
• An AI assistant deployed for internal employee use begins producing outputs that reference confidential merger discussions after being connected to the company's document management system. What type of AI risk does this represent?
• A risk officer is drafting an AI-specific addition to the organization's business continuity plan. Which scenario is MOST unique to AI system failures compared to traditional IT system failures?
• Which of the following BEST describes the risk introduced when an organization uses an AI model's output as a feature input to a second AI model in a pipeline?
• An organization's AI red team discovers that a deployed LLM can be made to ignore its system prompt and generate harmful outputs by prefixing requests with a specific multi-token sequence. What risk management response is MOST appropriate?