A utility team tried many outage models, checked the final holdout after every try, and now wants to quote that same holdout as an unbiased score of the winner. What is the proper use of a final holdout?

A final holdout estimates generalization only if it did not influence the choice. The proper use is to freeze the model using other data, then compute the holdout once and report that single figure. The utility team checked the holdout after every try, so those peeks already helped crown a winner, and quoting the best peek as an unbiased score overstates how the number was obtained. Training on the holdout after the search spends the set as fitting data and leaves no untouched report. Averaging the peeks into the training loss mixes the selection data with the final score and still uses the holdout to shape the result, so the peeks do not count as unused. The search can use validation splits inside the training pool as often as it needs. The final holdout is the set that stays sealed until the choice is frozen, and it is reported one time. Exam caveat: once a holdout has been consulted during selection, that same number is no longer an unbiased final report, however attractive the best peek looks. Operational check: document that the winning model was frozen from training folds alone, then compute the holdout a single time and archive that one figure as the report.