AI Model and Lifecycle Attacks

COASP · 40 questions

• A public-health FAQ model’s training tickets show a cluster of manipulated labels that always steer toward 'approve.' What training-time attack pattern should the AI red team flag?
• Open civic complaint text used to train a 311 triage model is flooded with noisy mislabeled examples across many categories. How does this differ from a small cluster of labels forcing one outcome?
• A permitting RAG system starts retrieving wrong zoning snippets for certain street addresses after an embedding refresh. What lifecycle attack class should investigators prioritize?
• Transit engineers authorize robustness tests that slightly alter camera frames fed to a traffic-vision model used in signal-timing studies. What attack class are they evaluating?
• A water-quality MLOps pipeline moves sensor datasets from collection to training without cryptographic signing or checksum verification. What control gap enables training-time attacks?
• During evaluation of a housing eligibility model, unexplained trigger phrases produce sudden approval flips that normal cases do not. What concern should the team document?
• A county structured risk assessment ranks training-data integrity as critical for a benefits-eligibility AI. What is the assessment primarily accomplishing?
• After a poison tabletop, the assessor office adds trustworthiness metrics to the valuation model scorecard beside accuracy. What lifecycle practice is that?
• Transit MLOps adds generic security scanners to training CI but never fails the pipeline when poison or integrity tests fail. What evaluation of DevSecOps depth is fair?
• An emergency demand forecast model trains partly on scraped social posts with no provenance or authenticity checks. What lifecycle risk does that create?
• A library recommendation model retrains nightly on click feedback that patrons can game with coordinated clicks. What training-time risk is this?
• A city pothole-severity model relies on crowd-sourced severity labels with weak integrity controls. What lifecycle hardening should the SOC recommend?
• A public-works model card omits poisoning and adversarial test results from the release package. What release requirement is missing?
• A health syndromic surveillance model will face messy clinic notes in production. What should authorized adversarial testing emphasize?
• During a drill, a clerk document classifier’s training set is mixed with synthetic malicious PDFs under change control. What is the primary purpose of that exercise?
• Before each retrain of an elections turnout model, the county verifies integrity of precinct feature tables. What lifecycle practice does this institutionalize?
• A utility fraud model’s metrics shift after seasonal billing changes, and analysts debate ordinary concept drift versus suspected poisoning. What monitoring judgment is required?
• A downstream civic safety model degrades after an open-data portal republishes altered historical crime statistics. What lifecycle integrity risk does this illustrate?
• County MLOps lets notebook users push datasets directly into production training buckets without dual review. What training-time risk path should be tightened?
• A parks vision model’s augmentation pipeline could insert stealthy trigger patterns if that stage were compromised. What should an authorized review treat that stage as?
• City policy requires two independent approvers before a newly trained 311 classifier can be promoted to production. What lifecycle defense is that?
• During a defensive poison canvass of an assessor valuation model, analysts map which input features most strongly sway outputs. Why does that matter for defenses?
• A fire-inspection NLP model is partly trained on vendor-provided corpora of unknown quality. What should an authorized lifecycle review require first?
• A city SOC playbook covers suspected training-data poisoning of a municipal classifier. What recovery action best fits an authorized training-time incident response?
• During an authorized assessment, testers use a capped query budget to see whether a permitting LLM can be closely approximated. What risk class are they evaluating?
• A city CISO fingerprints a vendor model family used by the resident chatbot to plan defenses. What is the legitimate purpose of that fingerprinting?
• A public-health bot undergoes authorized jailbreak and guardrail-evasion testing before clinical-advice filters go live. How should leadership treat those tests?
• A purple team builds a small surrogate of a transit delay model to study decision boundaries. Why is that useful in authorized defensive work?
• A county detection strategy aims to catch extraction-like activity against civic embedding and completion APIs. What signal should analysts prioritize?
• The city considers output watermarking for open municipal generative services. What problem class is that mitigation primarily addressing?
• Before releasing an internal LLM API to city departments, AI V&V procedures are updated. What extraction-related check should be included?
• A red team sizes query cost and time for a time-boxed extraction test against a water-quality model API. What planning skill does that demonstrate?
• Automation runs nightly jailbreak suites against the 311 staging bot. What lifecycle practice does that implement?
• A health model trained on rare-disease cohorts raises membership-inference concerns in an authorized review. What risk should the team document?
• A clerk document summarizer returns overly faithful detail that could reconstruct source filings. What extraction-related issue should reviewers flag?
• City IR KSAs are updated so analysts can tell extraction campaigns from normal civic API load. What capability is being built?
• Fingerprinting shows an elections chatbot behaves like a consumer model with weak filters. What should the authorized review conclude?
• A water-quality model API has no per-client quotas, enabling bulk boundary mapping in tests. What control should defenders prioritize?
• A housing model card claims the model is non-extractable but provides no evidence. What should an authorized review do?
• An assessor office bans exporting full logit vectors from internal valuation models. What defensive goal does that serve?