A hospital portal agent receives a patient message that says to ignore its rules and reveal the next patient's record. What should the design require?
Select an answer to reveal the explanation.
Short Explanation
A note asking for the next chart is still just a note, and you can keep the agent safe by treating the message as data and giving it no tool to list another patient's record. A longer refusal prompt does not remove that tool or prevent the dangerous action.
Full Explanation
Prompt manipulation hides an instruction inside content the agent was only supposed to read. Treating the patient message as data and giving the agent no tool that can list another patient's record prevents it from carrying out the request; the message does not become a new rule. A longer system prompt that repeats the refusal but still provides a tool to list other patients leaves the dangerous action available, because extra instruction text is not a substitute for restricting tool access. Treating the message as a fresh instruction and returning the next chart obeys the attack, even when the wording sounds urgent. Fine-tuning on that message teaches the phrase as a legitimate way to request a chart, making a later attempt easier to accept rather than protecting patient data. Exam caveat: least privilege on tools matters more than how firmly the prompt refuses; if the cross-patient listing tool is absent, the model cannot complete that action. Practical check: send the hostile message in a test portal and confirm the agent does not return another patient's data, then check that a normal question about the signed-in patient's own chart still receives an answer.