An AI product team is developing a generative AI system that will produce patient education materials for a hospital. The team proposes releasing the system without formal validation because 'it's just generating text, not making medical decisions.' How should the AI risk manager respond?

Requiring validation with specific focus on medical accuracy, hallucination rates, health literacy appropriateness, and potential harm from incorrect medical information is correct because patient education materials with incorrect medical information can directly harm patients. Patients may make incorrect health decisions based on AI-generated content about medication, symptoms, or treatment. The label 'text generation rather than medical decision-making' does not eliminate patient harm risk, even if the materials are not classified as clinical decisions. Validation must address medical accuracy, hallucination rates, comprehension, and health literacy appropriateness. Agreeing that text generation is low-risk and needs no formal validation ignores patient harm risk. Applying standard IT release management procedures because the system is ultimately just software is insufficient for AI systems with patient safety implications. Restricting the system to only non-medical administrative content to avoid validation eliminates clinical utility without addressing governance.