City clerks must decide if an agenda summarizer is ready for go-live. Which evaluation approach is most appropriate?
Select an answer to reveal the explanation.
Short Explanation
ROUGE tells you if the summary overlaps the agenda text; clerks tell you if it actually helps the meeting. Pair the automatic score with human-in-the-loop ratings before flipping go-live.
Full Explanation
City clerks deciding whether an agenda summarizer is ready for go-live need a robust municipal evaluation: pair automatic ROUGE on agenda summaries with clerk human-in-the-loop ratings. ROUGE checks overlap with reference agenda content; clerks judge whether the summary is usable for meeting prep—coverage of motions, clarity, and omission risk that overlap metrics miss. Both lenses together form a credible go-live gate for civic meeting support.
Shipping after a single perplexity printout with no human review fails by concept: perplexity measures token surprise, not agenda-summary usefulness to clerks. Using only GPU FLOPS as the go-live quality gate fails because compute throughput is unrelated to summary fidelity. Skipping automatic metrics and relying on anonymous social-media likes fails as governance: likes are biased, non-auditable popularity signals, not clerk task quality.
Exam caveat: define the clerk rubric (faithfulness, action-item coverage, neutrality) so HITL scores are comparable across reviewers and weeks. Operational check: compute ROUGE on the agenda holdout, collect clerk ratings on a stratified sample, and allow go-live only when both automatic and HITL thresholds clear versus the prior approved summarizer.