Foundations of Offensive AI Security
COASP · 22 questions
- A city CISO reviews a new 311 LLM assistant that answers resident requests and can open service tickets. How does this change the municipal attack surface compared with the prior static FAQ pages?
- A public-health chatbot pilot can answer symptom FAQs, retrieve clinic guidance, and call a scheduling plugin. Which description best captures the layered AI threats the city SOC should inventory?
- County permitting adds an AI “suggest documents” widget on the portal. The SOC wants an authorized offensive AI engagement. What should the scope definition emphasize first?
- A transit authority compares attacks on a rider-alert LLM with classic SQL injection on the same website. What distinction should the offensive AI lead emphasize?
- An emergency-management AI drafting tool loads third-party plugins for weather and GIS lookups. What risk should the municipal risk team flag first?
- City council staff ask how to frame a civic AI red-team charter using industry AI security references. Which approach best aligns municipal risk talks with common offensive AI frameworks?
- A water utility MLOps team asks how DoD-style AI ethical principles should shape authorized offensive testing of a plant-operations assistant. What is the soundest application?
- An assessor office wants assurance-case thinking before promoting a valuation-assistant LLM to public staff. What does that approach primarily require?
- A library director asks how switching from a self-hosted library chatbot to a vendor-hosted LLM changes the offensive testing plan. What is the key implication?
- A city AI working group is purple-teaming a permitting copilot. How should mission requirements shape offensive AI test objectives?
- A 311 agent uses RAG over knowledge articles. Which statement best explains why embeddings and retrieval are security-relevant architecture pieces?
- A public-health triage bot will be fine-tuned on clinic notes. How does fine-tuning most clearly change the municipal AI attack surface?
- A permitting assistant truncates long policy text mid-answer when the context window fills. Why is this an operational and security concern for offensive AI foundations?
- A fire/EMS dispatch assistant invokes tools for CAD lookups, while a parks FAQ bot only returns text. What architectural distinction should the diagram emphasize?
- A county clerk LLM pipeline runs moderation and logging stages before returning a reply. Why do those inference-pipeline stages matter to offensive and defensive testing?
- A housing authority bot stores short-term conversational memory across caseworker sessions. What foundational risk does this create?
- An open-data portal LLM wraps a vendor API. What should the attack-surface inventory include as LLM internals?
- A parks recreation chatbot plans multi-step facility bookings through tool invocation. Which foundational offensive AI concern applies most directly?
- A municipal FAQ model’s dataset curation workflow lacks integrity checks before training. What foundational assurance gap does this create?
- A city SOC defines knowledge and skills for staff who will red-team the 311 LLM versus classic web apps. Which foundational addition is most necessary for the AI track?
- A transit agency compares a self-hosted LLM on a city GPU cluster with a SaaS rider assistant. How should foundational recon approaches differ?
- An ethics board asks how assurance metrics help decide whether a permitting AI is trustworthy enough for public use. What is the best foundational answer?