A 311 chatbot team measures response quality by comparing generated answers against reference answers using BLEU and ROUGE scores. A reviewer notices these scores penalize a response that used different wording but conveyed the exact same information as the reference answer. What does this reveal about BLEU and ROUGE as evaluation metrics?
Select an answer to reveal the explanation.
Short Explanation
Think of grading an essay by counting how many exact words match a model answer key — a student who explains the same idea in their own words gets marked down even though they clearly understood it. BLEU and ROUGE work on that kind of word-overlap counting, so a paraphrase that says the same thing differently can score lower than it deserves. That's exactly why the reviewer is seeing a good answer get penalized.
Full Explanation
BLEU and ROUGE are lexical-overlap metrics: they compare generated text against one or more reference texts by measuring shared n-grams or word sequences, which means a response's score depends heavily on how closely its wording matches the reference rather than purely on whether its meaning is correct. This explains the reviewer's observation precisely, since a paraphrased answer that conveys identical information with different phrasing will have lower word overlap with the reference and can receive a lower score despite being just as correct. The option claiming these metrics evaluate factual accuracy independent of wording is the opposite of how they work; they are blind to meaning beyond what shared word sequences happen to capture, which is exactly why paraphrases get penalized. The option restricting them to chatbot tone evaluation misdescribes their origin and scope; both metrics originated in machine translation and summarization evaluation and are applied broadly to any generated-versus-reference text comparison, not narrowly to conversational tone. The option claiming no reference text is needed is factually backwards, since both metrics are fundamentally reference-based comparisons and cannot be computed without a reference text to compare against. Scope note: this is a known limitation, and teams needing paraphrase tolerance typically pair or replace these metrics with semantic-similarity approaches. Operational check: spot-check a sample of low-scoring responses manually to confirm whether low BLEU or ROUGE scores reflect genuine errors or just acceptable paraphrasing.