A city AI working group is purple-teaming a permitting copilot. How should mission requirements shape offensive AI test objectives?
Select an answer to reveal the explanation.
Short Explanation
Mission first: if the city exists to issue fair, accurate permits, your red-team hunts ways the copilot could misguide residents or yank the wrong tool—not just how hardened the cloud containers are. You won’t learn whether the copilot misguides residents by chasing public jailbreak leaderboard scores. And you can’t measure safety by perplexity and token fluency alone.
Full Explanation
Purple-teaming a permitting copilot should derive offensive AI objectives from municipal mission outcomes: incorrect permit guidance that harms residents or staff, data exposure, and unauthorized workflow or tool actions. Mission-mapped objectives keep effort prioritized on resident-facing harm and operational integrity rather than vanity metrics. Foundational practice links business and civic requirements to test goals so findings reflect impact the city actually cares about.
Prioritizing generic jailbreak benchmarks and public leaderboard scores can over-weight abstract prompt-breaking stunts while under-weighting permit-specific failure modes that actually endanger residents. Centering test objectives on cloud-hosting checklists and container-image hardening ignores that misleading permit guidance and unsafe tool actions are security and trust failures, not infrastructure posture issues. Judging success by model perplexity, token latency, and fluency improvements abandons the copilot’s AI safety surfaces entirely, since slick generation does not mean safe permit advice.
Exam caveat: derive AI test objectives from mission harm (wrong guidance, exposure, unsafe tools)—not generic jailbreak leaderboards, cloud hardening checklists, or raw fluency metrics alone.
Operational check: write objectives that name permit-misguidance and tool-abuse scenarios; score findings by civic impact; confirm the working group’s charter traces each objective to a mission requirement before execution.