A permit-desk QA bake-off used sampling; model A won by two questions on a 40-item set. What should happen before they retire model B?
Select an answer to reveal the explanation.
Short Explanation
Model A won by two questions on a 40-item sampled set. That gap can flip on a second pass. Repeat, add items, or run a paired test before retiring B. Trimming to A’s wins, or scoring sampler confidence, is not a lock.
Full Explanation
Small QA sets plus sampling noise can flip a two-item gap on a second pass. One lucky eval is not a retirement decision. A repeat, a larger item list, or a paired test shows whether the gap holds. Trimming the set to the winner's items, or scoring sampler confidence, manufactures a fake lock.