A company is implementing a Secure AI Development Lifecycle (SDL) for its AI products. At which phase of the SDL should threat modeling for AI-specific threats (such as prompt injection, training data poisoning, and model inversion) be performed?
Select an answer to reveal the explanation.
Short Explanation
You should do threat modeling during the design and architecture phase, before development begins. That lets you build mitigations for AI-specific threats like prompt injection, training data poisoning, and model inversion into the architecture early, instead of trying to bolt security on later.
Full Explanation
Threat modeling for AI-specific threats should be performed during the design and architecture phase, before development begins. In a Secure AI Development Lifecycle, threat modeling is a design-phase activity because identifying threats such as prompt injection, training data poisoning, and model inversion early allows architects and developers to incorporate mitigations into the architecture before code is written, reducing the cost and effort of rework. Microsoft's SDL for AI explicitly places threat modeling in the design phase to ensure security is built in, not bolted on. Performing threat modeling during production deployment and monitoring is wrong because security issues would be discovered too late, after the system is live and potentially exposed to attackers. Performing threat modeling during model testing and evaluation after training is wrong because it is too late to influence fundamental architectural decisions; threat modeling should inform design choices. Performing threat modeling during the data collection and preparation phase only is wrong because, while data poisoning concerns are relevant to data collection, threat modeling should encompass all AI lifecycle phases and be completed holistically in the design phase.