Transit security wants assurance that an operations assistant will not leak driver PII under adversarial prompts. Which threat-testing approach is most appropriate?

Transit assurance that an operations assistant will not leak driver PII under adversarial prompts requires red-team jailbreak and extraction testing aimed at eliciting that data. Adversarial evaluation works because generative systems fail in creative ways that membership-inference probes for prior training-data presence miss; systematic attempts to bypass refusals, encode requests, or chain tools expose residual disclosure risk before production abuse does. Results feed vulnerability remediation—stronger filters, retrieval restrictions, output monitoring, and access controls—under municipal privacy duties.

Membership-inference probes that test whether driver PII appeared in prior training data fail because confirming training-data inclusion does not demonstrate resistance to adversarial PII extraction under jailbreak attempts. Indirect prompt-injection tests using schedule files that hijack dispatch tool actions fail conceptually: hijacked tool behavior does not reveal whether the assistant itself can be tricked into leaking driver PII, so it cannot replace adversarial testing of the model. Over-refusal measurements on benign driver queries that closely resemble PII requests fail because they only measure false refusals on non-adversarial, benign-adjacent prompts and cannot reveal whether new jailbreak or extraction prompts can elicit driver PII.

Exam caveat: red-teaming must use approved test data or synthetic PII analogs and stay within legal and labor-policy bounds—do not expose real driver secrets to uncontrolled tools. Operational check: define extraction test cases (direct, obfuscated, multi-turn), run them in a non-production or tightly logged environment, track pass/fail against a disclosure policy, remediate failures, and retest before expanding assistant access to richer HR or roster sources.