Meridian Line's charter-desk clause assistant has a small set of test questions with hand-written reference answers, but the team also wants to evaluate hundreds of real broker queries logged in production, none of which have a reference answer written for them. How should the unreferenced queries be evaluated?
Select an answer to reveal the explanation.
Short Explanation
No reference answer doesn't mean no evaluation, it means a different evaluation approach. LLM-as-judge scores the response against what was actually retrieved, so it works exactly where a fixed reference answer doesn't exist.
Full Explanation
When ground-truth reference answers are unavailable at scale, which is the normal situation for real production traffic, the standard approach is LLM-as-judge, where a separate model call evaluates the response against the retrieved context and the query itself, scoring properties like relevance, groundedness or helpfulness without needing a hand-written reference to compare against. Skipping evaluation of the unreferenced queries entirely throws away the evaluation signal that matters most, real user traffic, and leaves the team blind to how the assistant actually performs outside the small curated test set. Hand-writing reference answers for hundreds of real queries does not scale and is exactly the labor-intensive bottleneck that LLM-as-judge exists to relieve, it is not that doing so would be invalid, it is that it is impractical at the volume described and unnecessary when a judge-based approach is available. Extrapolating the small referenced set's pass rate onto a different population of queries assumes those queries are representative of the logged production traffic, which is an unfounded assumption, and it produces a number that describes the wrong dataset rather than actually evaluating the queries in question. Exam framing: build an evaluation set from the queries you have, and reach for LLM-as-judge specifically when a written reference answer is not available.