The port-operations copilot answers open-ended planning questions from terminal planners, such as recommending which berth assignment best balances vessel arrival windows, questions with no single correct reference answer to compare against. How should the evaluation team score response quality for these questions?

LLM-as-judge scoring against a rubric is the standard technique for open-ended questions with no single correct output: instead of comparing the response to a fixed reference, the judge model evaluates specific qualities such as relevance to the question, actionability of the recommendation, and whether the response respects constraints stated in the prompt, then produces a score or pass/fail judgment per criterion. This preserves evaluation coverage without forcing every question into a closed form. Skipping evaluation for these questions entirely leaves the highest-value, hardest-to-get-right responses with no quality gate at all, which is backwards, since open-ended planning recommendations are exactly where mistakes are costliest. Computing BLEU against the most similar past response conflates similarity to prior text with correctness; a novel but excellent recommendation would score poorly simply for not resembling something already logged, and a bad recommendation that happens to phrase things similarly to a past good one would score well. Restricting the copilot to closed-form questions solves the evaluation problem by deleting the product requirement, and terminal planners specifically need open-ended berth-planning guidance, so that is not a real option. Rubric-based LLM-as-judge scoring is a core technique in Databricks Agent Evaluation for exactly this class of unreferenced, subjective task.