A team has finished all hyperparameter tuning and model selection using their training and validation sets. They now evaluate the final chosen model on a completely separate set of data for the first and only time. What is the main purpose of this last set?
Select an answer to reveal the explanation.
Short Explanation and Infographic
This is the test set, and its whole reason for existing is to stay untouched until the very end. Every decision about architecture, learning rate, when to stop — all of that gets made using the training and validation sets. The test set never influences a single one of those choices, so when you finally run the finished model on it, the resulting accuracy or loss is a clean, honest read on how the model will do on data it's never seen and never shaped its training around. That's why 'final, unbiased evaluation' is correct. It's not for more tuning — using it that way would contaminate it and inflate your confidence. It's not about rebalancing classes, that's a data-prep step done earlier. And it's never folded into training data — mixing them defeats the entire point of holding it out.
Full explanation below image
Full Explanation
The test set is a portion of data that is deliberately withheld from both the training process (the model never updates its weights based on it) and the model-development/tuning process (it is not used to make decisions about hyperparameters, architecture, or when to stop training — that is what the validation set is for). Its sole purpose is to be evaluated exactly once, after all modeling decisions have been finalized, to produce an unbiased estimate of how well the model will generalize to genuinely new, unseen data. Because it played no role whatsoever in shaping the model or its configuration, the performance metric obtained on the test set is not inflated by any form of tuning-related overfitting, unlike metrics obtained on data that influenced development decisions.
Using the test set to continue tuning hyperparameters such as the learning rate would defeat its purpose entirely. The moment test performance is used to make any decision about the model, it effectively becomes part of the tuning process (functioning like a second validation set), and any future evaluation on it would no longer be unbiased — this is a well-known pitfall sometimes called 'test set leakage.'
Rebalancing class distributions is a data preprocessing concern (such as oversampling a minority class, undersampling a majority class, or applying class weights in the loss function), typically addressed before or during training set preparation, and is unrelated to the role of a held-out test set.
Using the test set as extra training data is explicitly the wrong approach: if the training set is too small, the correct remedies are gathering more labeled data, applying data augmentation, or using techniques like transfer learning or cross-validation — never repurposing the held-out test data as training data, since doing so would eliminate the ability to get an honest final performance estimate.
The disciplined separation of train (fit weights), validation (tune and select), and test (final unbiased check, used once) is one of the most important practices in machine learning for producing performance claims that will hold up when the model is deployed on truly new data.