A rec-center QA set accidentally includes the same 'what is the pool fee?' items that were pasted as few-shot hints in the prompt. Why is the comparison contaminated?
Select an answer to reveal the explanation.
Short Explanation
The same “what is the pool fee?” items sat in the few-shot hints and on the scoreboard. That is leakage. Eval questions and supporting passages cannot live in the prompt or the training pile if the comparison is meant to measure unseen items. A one-word paraphrase does not clean it.
Full Explanation
A head-to-head QA score is meant to measure unseen questions. If the same items, or their supporting passages, appear as few-shot hints, the model is scored on material it already saw. That leakage inflates the comparison and must be rebuilt as a clean holdout. A one-word paraphrase or a 'prompts do not count' excuse does not restore an unseen test.