A housing agency tunes a model that predicts which vouchers will lapse. Analysts adjust parameters, check the final holdout after every change, and then report that holdout score as an unbiased estimate. What should they have done with the holdout instead?
Select an answer to reveal the explanation.
Short Explanation
You keep tasting the cake you promised to judge at the end, then you call that taste unbiased. Tune on your training folds and leave the final holdout alone until the voucher model and its parameters are frozen. Every peek you use to pick a setting spends the holdout.
Full Explanation
The final holdout estimates error on vouchers the search did not use. That estimate stays honest only if parameter choices are made without it. Tuning belongs on the training folds, where several splits can guide the search, and the holdout is scored once after the model and the parameters are frozen. Tuning on the holdout itself makes the reported score a best-of-many peek, not an unbiased estimate of a frozen model. Splitting that same holdout into folds and tuning there still spends the holdout; the best fold is then an optimistic pick from the data that was supposed to stay sealed. Skipping the training folds and selecting parameters from the holdout alone uses the final sample as the search space and leaves nothing untouched to report. Exam caveat: a training-fold score is the right guide during search and is still not a substitute for the one final holdout number once the model is frozen. Operational check: read the tuning log and confirm every comparison uses training-fold scores only, and that the holdout file is read once at the end.