Meridian Line's ML team wants to evaluate the retrieval component of its charter-desk clause assistant in isolation, before any language model touches the results, specifically whether the Vector Search index returns the charterparty clause chunks that actually contain the answer within the top results shown to the generator. Which metric class should they use?
Select an answer to reveal the explanation.
Short Explanation
Retrieval evaluation is a completely different job from grading the final answer, so it needs its own report card. Recall@k asks a simple question: of the clauses that actually mattered, how many did the index put in front of the generator? If you only score the finished text, a great retriever paired with a sloppy writer looks like two different problems wearing one costume.
Full Explanation
Recall@k and precision@k are retrieval metrics: they compare what Vector Search actually returned against a labeled set of clauses known to be relevant for each query, entirely independent of what the language model later does with those results. That isolation is the point here, since the team wants to know whether the index itself is finding the right material. BLEU compares two pieces of generated text against each other and was built for machine translation; comparing retrieved chunks to a generated answer conflates retrieval with generation and is not how BLEU is meant to be used. Perplexity measures how well a language model predicts a sequence of tokens, which speaks to fluency, not to whether the underlying content is correct or relevant, and it says nothing about the retriever. Token-level F1 against a reference answer is a generation metric that scores the final text output, so a perfect retriever paired with a poor summarizer would score badly there even though the index did its job. Keeping retrieval and generation metrics separate is exactly what lets a team pinpoint whether a bad answer traces back to the index missing the clause or the model mishandling a clause it was actually given.