Before deploying the customs-compliance agent, Nautilus Customs Brokerage's team needs a labeled evaluation set for its HS-code lookup tool calls, but writing a full hand-crafted reference answer for every possible customs question is too slow to keep pace with development. Which approach lets the team build a workable evaluation set fastest while still producing a meaningful quality signal?
Select an answer to reveal the explanation.
Short Explanation
You do not need a hand-graded answer key for every possible question to start evaluating, you need a solid core sample plus a judge that can grade the rest. Write real answers for the questions that matter most, then let an LLM-as-judge extend that coverage against a rubric instead of a reference, and you get useful signal in days instead of months.
Full Explanation
A practical evaluation set combines a small, carefully hand-labeled core, covering the representative question types the agent must handle, with LLM-as-judge scoring to extend coverage to a broader set of generated or collected questions where writing a full reference answer is not worth the cost. This gets an evaluation harness running quickly while keeping a trustworthy anchor of known-correct examples to sanity-check the judge against. Skipping an evaluation set and relying only on ad hoc production feedback removes any repeatable, pre-deployment quality gate and means every regression is discovered by a live customs officer instead of a test run. Reusing the embedding model's training set is a data-leakage trap: that data was used to shape the model's own behavior, so scoring against it measures memorization rather than generalization, and it likely was not built to test the agent's tool-calling correctness at all. Waiting six months for organic traffic to accumulate blocks the team from evaluating anything before the first deployment, which defeats the purpose of pre-deployment evaluation entirely and delays catching problems that a small curated set could surface immediately. The curate-plus-judge approach is also exactly the pattern Databricks recommends for Agent Evaluation workflows where full reference answers do not scale.