A clerk's office is choosing a foundation model and has both vendor benchmark scores and sample summaries on local ordinances. Which evaluation approach is most appropriate?
Select an answer to reveal the explanation.
Short Explanation
Vendor scoreboards are like racing posters—nice, but not your commute. Run the candidates on the clerk's real ordinance packets and judge summarization quality there. Local task benchmarks beat brochure metrics.
Full Explanation
A clerk’s office choosing a foundation model with both vendor benchmark scores and sample summaries on local ordinances should evaluate FMs with municipal task benchmarks on real ordinance summarization quality, not marketing metrics alone. Vendor leaderboards measure broad capabilities under someone else’s prompts and data; clerk packets have local structure, citation norms, and risk tolerances that only appear when you score candidates on your own samples. The appropriate approach blends published scores as a shortlist filter with hands-on evaluation on ordinance summarization.
Selecting solely from marketing leaderboards without testing on local ordinance packets risks buying a model that looks strong globally but summarizes poorly on municipal legal prose. Picking the model with the flashiest product video regardless of summarization accuracy substitutes branding for evidence. Avoiding any local testing because vendor averages always transfer to court packets is an invalid assumption for specialized civic documents.
Exam caveat: evaluation questions emphasize task-specific benchmarks and human review on representative data over brochure metrics alone.
Operational check: assemble a labeled set of ordinance packets and reference summaries, score candidate Bedrock models on faithfulness and usefulness, include clerk reviewers, and document why the winner was chosen beyond vendor chart position.