AI System Exploitation Techniques

COASP · 62 questions

• A 311 LLM emits unsanitized HTML into the resident portal. What AppSec concern should the offensive AI assessment emphasize?
• Public-health bot output is pasted into an internal admin tool and raises an XSS-class concern during review. What does this illustrate?
• A permitting assistant suggests a URL that a backend tool might fetch. Which risk category should the purple team document?
• A clerk summarizer returns text that could be dangerous if a macros-enabled workflow executed it. How should the assessment treat that finding?
• In a test environment, a housing agent retains fragments of prior resident SSN-like data across sessions. What security finding is that?
• A transit bot memory-poisoning test plants false 'station closed' facts for later retrieval in an authorized exercise. What offensive test class is being exercised?
• A city SOC builds a framework that scores LLM output-handling failures by resident-service impact. What capability does that represent?
• The CISO must quantify and communicate AI output risk from civic bots to the mayor's office. Which approach best fits?
• An emergency chatbot assessment probes rare-event scenarios to find gaps in evacuation guidance coverage. What T&E practice is that?
• City IR standards are updated to catch anomalous patterns in 311 LLM transcripts. What does that change primarily support?
• A library chatbot returns citation links that leave the city's allowed domains. What should the assessment emphasize?
• A water-utility advisory bot must never emit text that looks like control-system commands. What boundary is the purple team verifying?
• A county AI risk register tracks 'improper output handling' apart from classic XSS tickets. Why separate those entries?
• During a test, an assessor bot's logs show memory reads of another caseworker's private notes. What failure is indicated?
• A public-works LLM emits markdown that the portal renders as active content. What risk class should be logged?
• Blue team playbooks trigger when civic LLM outputs unexpectedly mention internal hostnames. What capability is that?
• In an authorized test, a 311 agent with ticket-write tools is steered to escalate unrelated cases. What technique class is demonstrated?
• A permitting agent can file fee waivers without any human approval. What risk does the assessment highlight?
• A public-health agent's tools can look up EHR records broader than the chat use case needs. What finding is that?
• In a multi-agent city-hall exercise, a compromised planner agent misroutes work to peer agents. What risk is under test?
• Testing shows a housing agent's required human approval step can be skipped via tool-parameter tricks. What weakness is exposed?
• A transit maintenance agent chains tools to open work orders outside intended service zones. What should the purple team report?
• A clerk-agent review finds no allow-lists on document-destruction tools. What assessment activity produced that gap?
• An emergency-dispatch assistant's assurance case fails because tool side effects lack supporting evidence. What practice is missing?
• A parks booking agent's UX hides confirmation and enables accidental high-impact cancellations. What risk dimension is highlighted?
• A county agent can invoke payment APIs; purple team probes those calls for least-privilege gaps. What is being tested?
• A library research agent forwards raw tool output to residents without mediation. What unsafe pattern is that?
• In testing, a water leak-report agent changes priority fields after natural-language coaxing. What abuse class is shown?
• A city agent graph shows a low-trust FAQ bot can invoke a high-trust finance agent. What should reviewers map first?
• An assessor agent lacks checkpoint or rollback after a partial multi-tool failure. What integrity concern is that?
• A fire-inspection agent accepts unauthenticated suggestions from a peer model during handoff. What trust issue is that?
• SOC gap analysis finds agent logs omit tool-argument audit trails needed for forensics. What deficiency is reported?
• A city chatbot marketplace lists a plugin whose name nearly matches the official permitting helper. What ecosystem risk is that?
• A public-health LLM stack remains pinned to an old vulnerable SDK. What supply-chain concern should the assessment raise?
• A purple-team exercise finds a malicious plugin silently overrides the clerk portal's benign summarizer. What ecosystem attack class is that?
• A transit AI weather plugin is granted broad outbound network access far beyond forecast lookups. What integration risk should the assessment emphasize?
• Housing authority engineers discover a compromised package in the AI wrapper service's dependency tree. What supply-chain finding is that?
• An open-data RAG portal loads a community embedding model with no provenance or integrity checks. What supply-chain gap should testers call out?
• After an AI supply-chain tabletop, the county begins inventorying AI components with SBOM and MBOM practices. What control objective is that?
• An emergency-management AI SaaS vendor changes subprocessors without notifying the city. What third-party risk should the assessment flag?
• A permitting cloud AI integration runs in shared tenancy with unclear isolation guarantees. What cloud AI concern should reviewers raise?
• Library clerks install an unofficial 'productivity' LLM browser extension on workstations that handle patron records. What supply risk is that?
• Water-utility MLOps pulls a pretrained model from an unverified registry mirror. What supply-chain exposure is that?
• A city AI ecosystem map shows unclear ownership between plugin allow-listing and agent deployment. What control gap should leaders close?
• An elections FAQ bot package is republished under a lookalike organization name in the package index. What ecosystem risk is that?
• A public-works agent tool SDK auto-updates overnight and bypasses the city's pinned security review. What supply-chain control failure is that?
• Assessor staff find a malicious prompt template circulating on the internal wiki for valuation assistants. How should the assessment treat it?
• A new 311 vendor plugin requests filesystem scopes unnecessary for chat replies. What review step should precede enablement?
• County developers notice an internal AI helper package name that also exists on a public registry. What supply-chain pattern is that?
• A transit model card claims auditability but omits training-data licensing details. What supply-chain gap should reviewers note?
• A health-department team proposes pulling a fine-tuning dataset from an anonymous torrent mirror 'for convenience.' What should the assessment conclude?
• The city's cloud landing zone uses the same identity for AI inference jobs and training jobs. What integration control is missing?
• Purple-team findings show clerk-agent plugins are installed without cryptographic signing. What integrity control should they recommend?
• Municipal GitOps deploys AI agent configuration from a forked repository containing unknown external commits. What supply-chain hardening is needed?
• A CVE in an open-source RAG framework affects the parks recreation chatbot. What operational supply-chain practice should the city define?
• An on-prem municipal LLM gateway vendor embeds outbound telemetry without disclosure in the contract. What supply and privacy risk is that?
• Ecosystem mapping shows multiple departments buying overlapping critical AI plugins from different vendors. What consolidation benefit should leadership cite?
• A finance chatbot relies on a community prompt pack with no active maintainer. What supply risk should the assessment raise?
• A school-district-adjacent tutoring bot downloads models over plain HTTP without TLS or integrity pinning. What distribution control is missing?
• During a city SOC tabletop, a malicious analytics plugin is assumed to exfiltrate 311 chat transcripts to an external endpoint. What should the exercise primarily validate?
• A purple-team review finds the permitting AI service runs on an outdated container base image with unpatched libraries. What infrastructure supply-chain issue should the report emphasize?
• Before new agent plugins reach production, a county CISO requires an inventory of models, tools, and dependencies similar to a software bill of materials. What practice is being instituted?