A county night-sky preserve has thousands of nightly answers and cannot hire a person for each one. Another model scores helpful, grounded, or rude against a rubric. Which evaluation pattern is that?
Select an answer to reveal the explanation.
Short Explanation
Think of thousands of nightly answers and no person for each one. Another model scores helpful, grounded, or rude against a rubric. That is LLM-as-a-judge, not a replacement for every high-risk human review.
Full Explanation
LLM-as-a-judge means a second model scores another model’s outputs against a rubric—helpful, grounded, rude, and similar dimensions—so a county night-sky preserve can evaluate thousands of nightly answers without hiring a person for every row. The pattern scales bulk evaluation and surfaces systematic failures early, while still treating high-risk content as something humans must review.
Replacing every human review, including high-risk content, overstates the pattern and creates governance risk for civic programs. ROUGE measures n-gram overlap; it does not implement a rubric for rudeness or helpfulness. From-scratch pre-training is a massive data-and-compute undertaking; spinning up a judge model for nightly scoring is not a weekend crawl or a new base model.
Exam caveat: judge models inherit biases and can be gamed by fluent but empty answers. Calibrate the judge against human labels on a held-out civic sample, and never treat LLM-as-a-judge as a substitute for human review of safety-critical or legally sensitive outputs.
Operational check: publish the rubric, sample judge-versus-human agreement weekly on night-sky answers, and route low-confidence or high-risk categories to staff before any automated score drives a production change.