A public-health LLM project summarizes outbreak briefs and also classifies triage intents. How should metrics be chosen?
Select an answer to reveal the explanation.
Short Explanation
Summaries and triage labels aren’t the same homework. Score the brief with ROUGE overlap and the triage tag with accuracy—right yardstick for each phase of the health project.
Full Explanation
A public-health LLM project that both summarizes outbreak briefs and classifies triage intents must pick metrics that match each task head. ROUGE (or similar overlap metrics) scores how well summary text covers reference brief content; classification accuracy (or F1) scores whether triage intents land in the right label buckets. Using the right yardstick per phase keeps the civic program honest: a model can be a good summarizer and a bad triage classifier (or the reverse), and blended or mismatched metrics hide that split.
Using only GPU temperature sensors as the sole quality metric fails by concept: hardware telemetry measures heat, not summary faithfulness or triage correctness. Applying BLEU exclusively to both summarization and classification fails because BLEU is an n-gram overlap metric for generated text, not a proper score for discrete intent labels—forcing it onto classification misreads task success. Skipping metrics until after production incidents accumulate fails as risk management: public-health systems need pre-production gates, not post-harm discovery.
Exam caveat: ROUGE can miss clinically dangerous omissions that still overlap many tokens—pair summary metrics with clinician review on a sample. Operational check: report ROUGE on the summary eval set and accuracy/F1 on the triage set from the same training run, set independent promotion thresholds for each phase, and block go-live if either civic metric regresses versus the prior approved checkpoint.