A school IEP summarizer improves BLEU but clinician Likert scores fall. What should the team prioritize for stopping and promotion?
Select an answer to reveal the explanation.
Short Explanation
Fancy overlap scores aren’t the same as a clinician saying the IEP summary is usable. If Likert scores drop while BLEU climbs, toss that run—kids’ plans need human judgment in the loop.
Full Explanation
When a school IEP summarizer improves BLEU but clinician Likert scores fall, stopping and promotion must prioritize the human civic metric. Reject runs that worsen clinician Likert even if BLEU rises. Automatic n-gram overlap can climb while summaries become less usable, less faithful to goals/services, or less clinically coherent—exactly what district clinicians score on Likert rubrics. Kids’ plans need human judgment in the promotion gate, not BLEU theater.
Promoting solely on the BLEU gain and ignoring clinician scores fails by concept: BLEU is not a clinical usability certificate. Averaging BLEU with disk free-space percentage as a composite fails because storage telemetry is unrelated to summary quality and dilutes the clinician signal with noise. Disabling human scoring because automatic metrics are sufficient fails for high-stakes education content: automatic metrics routinely miss omissions and tone issues clinicians catch.
Exam caveat: define Likert sampling size and inter-rater process so human scores are stable enough to gate training—tiny n makes early stop jittery. Operational check: each candidate run, compute BLEU and mean clinician Likert on the same IEP holdout; auto-reject if Likert drops versus baseline regardless of BLEU; promote only when Likert holds or improves and secondary automatic metrics stay within agreed bounds.