A county purple team documents prompt patterns that try to jailbreak a public LLM into providing weaponized code. What should policy do?
Select an answer to reveal the explanation.
Short Explanation
Jailbreak prompts that beg for weaponized code are a red flag, not a staff sport. County policy should spot and block that risk—not sticky-note the winning strings on the intranet.
Full Explanation
County purple teams that document prompt patterns attempting to jailbreak a public LLM into weaponized code should treat those patterns as LLM guardrail-bypass risk. Acceptable-use policy should block them: staff must not practice or share jailbreaks that seek harmful offensive code outside explicitly authorized lab programs with separate controls. Recognition-plus-block is the policy action that protects municipal AI tenants and clarifies that 'research' framing does not automatically authorize bypass attempts.
Publishing successful jailbreak strings on the intranet for 'research sharing' spreads bypass tradecraft to the wrong audience and increases misuse risk. Requiring weekly jailbreak practice on production AI tenants normalizes policy violation on systems that may hold or process county data. Ignoring jailbreaks because 'public LLMs cannot produce harmful content' is a false assurance—guardrails exist precisely because harmful emissions are a risk.
Exam caveat: policy should recognize and block guardrail-bypass patterns, not celebrate, mandate, or dismiss them.
Operational check: add jailbreak/weaponized-code prompt patterns to county AI monitoring and DLP/AUP enforcement, train staff on authorized lab-only paths for offensive research, and route detected attempts to security for coaching or discipline rather than intranet how-to posts.