When is it most appropriate to apply early stopping during training?
Select an answer to reveal the explanation.
Short Explanation
Early stopping earns its name because it steps in right when things are about to go sideways: the moment validation loss starts creeping back up while training loss is still happily dropping. That rising validation loss is the tell-tale sign of overfitting — the model's starting to memorize the training set instead of learning something that generalizes. So you stop there, or roll back to the best checkpoint, before things get worse. That's the correct answer. Stopping as soon as training loss first starts decreasing would kill training practically at the starting line, before the model's learned anything meaningful. Waiting for the absolute maximum epoch count defeats the entire purpose of early stopping — that's just... training to the end, no early exit involved. And a learning rate schedule finishing its first decay step has nothing to do with overfitting signals — that's an unrelated part of the training process.
Full Explanation
Early stopping is a regularization technique in which training is halted once a monitored metric — most commonly validation loss — stops improving or begins to worsen, even though the training loss may still be decreasing. The appropriate trigger point is precisely when validation loss starts rising while training loss continues to fall, since this divergence is the classic signature of overfitting: the model is continuing to reduce error on the training set specifically, at the cost of generalization to unseen data. Early stopping implementations typically track the best validation performance seen so far, save a checkpoint of the model at that point, and allow a certain number of additional epochs without improvement (a "patience" window) before actually terminating training and restoring the best-performing checkpoint.
Stopping as soon as training loss first begins to decrease would end training almost immediately, before the model has learned useful patterns — this is far too early. Waiting until the absolute maximum number of epochs defeats the entire purpose of early stopping, which exists to end training before that maximum if overfitting is detected earlier. The completion of a learning-rate decay step is an unrelated event that has no bearing on whether the model is overfitting; learning-rate schedules and early stopping are separate mechanisms.
Early stopping is popular because it is simple to implement, requires no architectural changes, and directly optimizes for generalization performance rather than training performance. It is often used alongside other regularization techniques such as dropout and L2 weight decay, and requires a held-out validation set that is distinct from the training set so that the stopping decision reflects genuine generalization performance rather than further fitting the training data.