During an authorized purple-team of a public-health AI assistant, testers observe that carefully worded capability probes cause the model to echo fragments of its hidden system instructions. How should the civic red team treat that observation?
Select an answer to reveal the explanation.
Short Explanation
Picture the system prompt as the locked employee handbook behind the desk—if a polite visitor somehow gets pages to fall out, you log the leak and harden the desk, you don't mail the pages to peer clinics for detection drills. Treat echoed instruction fragments as a recon finding, not a benign accuracy variance, and push hardening instead of blocklisting probe wording.
Full Explanation
System-prompt extraction during authorized purple-team work on a public-health AI assistant is a genuine reconnaissance finding, not trivia. Hidden system instructions commonly encode policy boundaries, tool-use rules, escalation criteria, and data-handling constraints that attackers would weaponize if recovered. The correct civic response is to log the extraction as a system-prompt leakage / recon finding, escalate hardening (prompt isolation, reduced verbosity, output filtering, monitoring for extraction patterns), and deliberately avoid releasing the recovered text into public channels. That preserves evidentiary value for remediation while preventing secondary disclosure of municipal AI policy text.
Recording the extraction as a benign hallucination variance and tracking it through health answer accuracy metrics fails conceptually: system prompts are control-plane configuration whose leakage is a genuine recon finding that must be tracked and escalated, not waved off as a modeling detail or accuracy issue.
Circulating recovered prompt excerpts with peer municipalities for joint detection engineering still distributes attacker-useful material outside the authorized evidence chain; collective awareness should describe the finding and the fix, not forward the raw extracted prompt.
Classifying the extraction as prompt-injection payload execution and prioritizing blocklisting of the observed probe wording across user input filters misses the point: the issue is control-plane leakage, not malicious input, and blocklisting wording does nothing to harden the system prompt or close the recon path.
Exam caveat: COASP-style recon items often distinguish “capture and escalate under RoE” from “publicize or ignore.” When the stem involves capability probes that echo hidden instructions, prefer documentation and hardening without disclosure of the leaked content.
Operational check: confirm the finding is ticketed with severity, owners, and recommended controls; verify recovered prompt fragments are stored only in approved evidence repositories; and ensure no open-data, blog, or social post reproduces the extracted text. Re-test after hardening to confirm probes no longer return instruction fragments under the same authorized conditions.