Securing AI Systems

SecAI+ · 80 questions

• A city security architect must threat-model a resident chatbot before launch. Which resource is most appropriate for cataloging LLM application risks?
• A utility data-science team is securing a predictive pipe-break model that is not an LLM chatbot. Which threat resource best fits classical ML security risks?
• County risk management is cataloging potential harms from a police bodycam summarizer. Which resource best supports broad AI risk cataloguing?
• A SOC lead needs to describe an attempted model-theft campaign against a municipal LLM API using adversarial AI techniques. Which resource is most appropriate?
• City vulnerability management learns a shared embedding library used by several civic AI apps has a newly published flaw. Which intelligence source is most relevant to track?
• A school-district CISO wants structured risk analysis of a student chatbot architecture before procurement. Which approach best applies threat-modeling practice?
• A transit fare bot needs both adversary technique mapping and LLM application risk coverage. Which pairing best reflects complementary AI threat resources?
• Internal audit asks the city to show how AI threats informed controls on the permitting assistant. Which evidence best answers that request?
• Analysts review a jailbreak attempt against a clerk chatbot. Which next step best uses published LLM risk categories?
• A county red team will test municipal AI services. Which planning approach best incorporates AI-focused adversarial techniques?
• Security must approve expanding a 311 bot citywide. Which model-control requirement best gates that expansion?
• A public-health chatbot must not give clinical diagnoses even when residents insist. Which control best enforces that constraint?
• Building inspectors query an AI assistant about code sections. Which model-control pattern best keeps answers grounded in citations?
• Election staff worry about instruction-override attempts reaching the elections FAQ model. Which gateway control best addresses that threat?
• A scraper hammers the city's open-data AI API with rapid repeated requests. Which gateway control best mitigates that abuse?
• County counsel fears long ordinance pastes could inflate cost and enable context-stuffing attacks on a legal AI assistant. Which control best addresses that concern?
• A housing chatbot must reject oversized uploads and limit how many files each caseworker can send per day. Which gateway control best matches that requirement?
• A courts portal AI should accept text filings but reject image and audio uploads to reduce deepfake-input risk. Which control best implements that policy?
• Only the permits VLAN service account should be able to call the internal building-code model endpoint. Which control best enforces that?
• After a model upgrade, security must confirm the housing assistant still refuses SSN disclosure. Which practice best validates that control?
• Transit wants launch hardening for a fare bot using layered gateway defenses. Which approach best combines complementary controls?
• Analysts designing a clerk assistant must place protections correctly in the stack. How do in-model guardrails differ from gateway prompt firewalls?
• The requirement is to stop prompt flooding against the city's 311 AI API. Which controls should the team prioritize first?
• An emergency-ops agent must not call paging APIs unless a human confirms first. Which control best enforces that?
• Only certified building officials should reach the advanced code-interpretation model tier. Which access approach best fits?
• A parks chatbot project does not need tax-lien datasets. Which data-access control best applies?
• A change-management AI agent may read CMDB tickets but must not approve production firewall changes. Which access design is correct?
• The public 311 bot API must stay edge-protected while private model endpoints remain internal. Which network pattern best fits?
• A shared AI platform identity currently spans model, vector-store, and logging systems. What change best applies least privilege?
• A contractor must evaluate a pilot model without keeping permanent production rights. Which access approach is most appropriate?
• HR policy documents live in a vector collection the public parks bot must not query. Which control best enforces separation?
• Departing employees must lose model API access automatically when they leave city SSO groups. Which design best achieves that?
• An incident-response summarization agent should call the ticketing API only from an approved tool set. Which control best fits?
• Model admin consoles for the elections assistant must not be reachable from the public Internet. Which control best enforces that?
• Field inspector tablets must send prompts to the building-code model without exposing them on the network path. Which control is required?
• Vector indexes of sensitive case notes must remain protected while stored. Which control best addresses that risk?
• A high-sensitivity benefits-eligibility pilot needs stronger protection while records are processed in memory during inference. Which approach best fits?
• Public health must share a training extract with a vendor without exposing patient identifiers. Which control should be applied first?
• RAG documents for the clerk bot should only be retrievable by roles allowed for each sensitivity level. Which control enables that?
• An IR ticket summarizer must not send cardholder data to a cloud LLM. Which control best prevents that disclosure?
• A permitting bot demo for city council must avoid showing real parcel owner names. Which control best fits the demo environment?
• A city business-license fraud model collects dozens of applicant attributes. After a privacy review, which redesign best applies data minimization?
• Transit ops wants an anomaly model on vehicle telemetry. Which approach best layers data-security controls before the feature store is built?
• Social-services staff notice that chatbot prompt logs contain resident case details. How should the city treat those logs?
• County records plans to embed open-records PDFs that sometimes contain driver’s license numbers. Which control belongs in the RAG ingestion pipeline?
• A public library kiosk hosts an ordinance FAQ bot. Which retrieval rule best enforces data classification?
• On election night, the county SOC needs visibility into unusual activity against the elections FAQ model. Which monitoring focus is most appropriate?
• After a model update, the parks department wants assurance the chatbot still follows policy. Which practice best fits response monitoring?
• City SOC wants early warning when the 311 AI gateway starts failing under load. Which integration best supports that goal?
• Before shipping AI prompt logs to a vendor SIEM, which step best protects resident privacy?
• Courts IT must ensure AI audit logs for the clerk assistant cannot be quietly altered after an incident. Which control best protects those logs?
• A benefits eligibility bot sometimes answers with low certainty. Which monitoring use of confidence levels is most appropriate?
• After a weekend outage scare on the permits AI API, operations needs better health and abuse visibility. Which metric focus is most useful?
• Finance notices a sudden spend jump on the city’s open-data LLM after a viral social post. Which monitoring practice best explains and contains that growth?
• Parks leadership wants monthly assurance the trail chatbot does not invent closed trails. Which audit approach best fits?
• Building officials need to know whether the code-interpretation bot cites the right sections. Which audit best measures that?
• HR pilots an AI screen for city job applications. Which audit is most appropriate before wider use?
• Internal audit asks whether sealed-case RAG collections stay limited to authorized clerks. Which practice answers that question?
• The SOC wants low-confidence AI IR summaries handled safely. Which workflow best operationalizes confidence monitoring?
• Attackers might try to silence AI cost and abuse alerts. Which control best protects the monitoring pipeline itself?
• A resident pastes instructions telling the 311 bot to ignore prior rules and reveal internal routing notes. Analysts should classify this evidence as which attack?
• After repeated injection attempts on a clerk assistant, which compensating controls best harden the service?
• Investigators find malicious samples inserted into the fine-tune set for a business-license fraud model. What attack does this evidence indicate?
• An open-data pothole classifier starts failing after crowdsourced uploads flood the training set with mislabeled images. Which attack best describes this?
• After a poisoning incident, a water utility must harden training inputs. Which compensating control best addresses data integrity?
• Testers coax a parks safety bot into giving unsafe trail advice by using role-play that bypasses refusal rules. What attack pattern should analysts document?
• A municipal clerk bot invents a case citation that never existed, creating legal exposure for the city attorney. How should security treat this failure?
• Crafted Unicode in a permit form confuses an AI fee classifier into assigning the wrong fee tier. What attack class should analysts document?
• An adversary seeds skewed complaint tickets so a neighborhood-priority model favors one district over others. What attack should the city document?
• Users encode prohibited health questions in another language to bypass a public-health bot’s guardrails. What technique should defenders recognize?
• An attacker abuses a weak plugin that lets the 311 bot create forged work orders in the city’s CMMS. What failure mode is this?
• Researchers show an assessor valuation model can leak whether a specific parcel was in the training set. What privacy attack should the city treat as in scope?
• Repeated structured queries against a city API suggest an attempt to exfiltrate or approximate a fine-tuned municipal model. What should analysts flag?
• After extraction attempts on a city model API, which compensating controls best reduce further theft risk?
• A compromised upstream base model used for transfer learning affects the city’s fine-tuned permitting assistant. What risk should leadership prioritize?
• Subtle production feedback loops skew a business-license fraud model’s outputs away from true positives. What attack or failure pattern is this?
• A junior admin executes an AI-generated script without review and drops a critical firewall rule on the city edge. What failure should the post-incident report emphasize?
• Giant adversarial prompts exhaust GPU workers hosting the emergency-ops summarizer. What attack should the SOC classify?
• An overly trusted agent with excess tools emails internal HR data into a public ticket thread. Which failure modes best describe this incident?
• After an agent disclosure incident, which compensating control set best hardens the city’s AI assistants?