A team builds an agent that generates inline code comments. They evaluate it using BLEU score against a reference set of human-written comments, achieving a score of 0.72. When they deploy the agent, developers report the generated comments are technically accurate but unhelpful — they often restate what the code does rather than explaining why. The BLEU score does not capture this quality dimension. What does this reveal about the team's evaluation approach?
Select an answer to reveal the explanation.
Short Explanation and Infographic
BLEU score measures how closely the generated text matches a reference — but it can't tell you if the comment is actually useful to a developer. High BLEU on unhelpful comments means the metric is measuring the wrong thing. The evaluation must be aligned to the real goal: does this comment help a developer understand the code's intent?
Full explanation below image
Full Explanation
BLEU (Bilingual Evaluation Understudy) score is a metric from machine translation that measures n-gram overlap between generated text and reference text. It is a proxy for similarity, not a direct measure of utility. A comment can closely resemble the reference while still failing to convey the reasoning, design decisions, or intent that developers actually need.
Metric-goal misalignment occurs when the chosen metric is technically measurable but does not capture the actual quality dimension that matters for the use case. For code comments, developer utility requires understanding of: - Why this code was written this way (design rationale) - What edge cases it handles and why - What the reader should know to safely modify this code
Better-aligned metrics for this use case include: - Developer satisfaction ratings (1-5 stars per generated comment) - Rubric-based evaluation scoring whether the comment addresses 'why' vs. 'what' - A/B testing measuring developer task completion speed when comments are present - Human-in-the-loop evaluation with specific criteria around explanatory depth
Option A (higher BLEU threshold) is incorrect because the problem is not with the score's threshold but with what the score measures. No BLEU score value can distinguish 'what' comments from 'why' comments.
Option C (reference set too small) would affect statistical reliability of the score but would not explain the systematic gap between high BLEU and low developer utility.
Option D (longer comments) addresses surface-level length but not content quality. A longer comment that still restates the code without explaining intent would score well on BLEU and still be unhelpful.