A public-works office wants to know whether this week's checkpoint…

Question

A public-works office wants to know whether this week's checkpoint beats last month's on 'which pump tripped?' What makes that head-to-head valid?

Accepted Answer

Lock the same gold questions, the same context windows, and the same scoring script for both checkpoints. — This week’s checkpoint versus last month’s on “which pump tripped?” Lock the same gold questions, the same context windows, and the same scoring script. Favorite question lists, a longer window for the new run, and mixed hand-versus-script scoring manufacture a fake win.

Answer

Let each checkpoint use its favorite questions and its own scorer, then compare the two headlines.

Answer

Change the context window only for the new checkpoint so it has a better chance.

Answer

Score the new checkpoint by hand and the old one automatically, because mixed methods are fairer.

A public-works office wants to know whether this week's checkpoint beats last month's on 'which pump tripped?' What makes that head-to-head valid?

Short Explanation

Full Explanation