A permitting office runs a hybrid pipeline: a traditional classifier first sorts an application into a permit type, then a foundation model drafts a plain-language explanation for the applicant. How should the team evaluate this pipeline's performance?
Select an answer to reveal the explanation.
Short Explanation
A classifier and a generator do different jobs, so grading them with the same ruler doesn't tell you much. The classifier's job is picking the right bucket — that's a precision-and-recall question — while the FM's job is writing usable prose, which needs its own kind of check. Give each stage the evaluation it actually deserves.
Full Explanation
Traditional classification and generative drafting fail in different ways, so they need different measuring sticks: precision and recall (or a confusion matrix) tell you how often the classifier assigns the correct permit type, while GenAI evaluation techniques — human review, an LLM-as-a-judge rubric, or NLP similarity metrics — tell you whether the generated explanation is accurate, clear, and appropriately toned. A single blended score hides which stage is actually failing when quality drops, since a strong classifier paired with a weak explanation, or the reverse, both wash out into one number. Scoring only the generated text ignores that a perfectly written explanation of the wrong permit type is still a failure the applicant experiences directly. Applying BLEU to classification labels misapplies a text-overlap metric designed for comparing generated language against reference language to a task that produces discrete categories, where it has no meaningful interpretation. Scope note: keep the two evaluation pipelines separate in dashboards so a regression in one stage doesn't get masked by the other. Operational check: periodically sample cases where the classifier was correct but the explanation was flagged poor, to confirm the split is catching stage-specific issues.