While training the ramp-safety camera model, the Meridian data science team's learning curves show training error still falling after every pass through the data, while error on the held-out validation set bottomed out several passes ago and has been rising since. The team asks the PM whether to let the run continue to its scheduled end. What should the PM support?
Select an answer to reveal the explanation.
Short Explanation
When your training error keeps falling but validation error has turned upward, the model has started memorizing — stop at the validation low point and keep that checkpoint.
Full Explanation
When training error keeps falling but error on held-out data bottoms out and then rises, the model has begun to overfit: later passes fit noise and quirks of the training frames instead of patterns that carry over to new ramp scenes. The standard response, often called early stopping, is to halt near the validation minimum and retain the model saved at that point, which gives the best expected performance on camera footage the system has never seen. Letting the run finish because training error is still improving is wrong because training error alone says nothing about generalization; the widening gap is the warning sign, and more passes only widen it. Folding the validation set into the training data is wrong because it removes the only independent signal the team has, so the reported accuracy would be inflated and could no longer be trusted at the Go/No-Go review. Scheduling more passes on the theory that the model is underfit is wrong because underfitting shows up as poor error on both training and validation data, not as a validation curve that is rising while training error improves. A PM does not tune the run personally, but needs to recognize this pattern and back the decision that protects real-world safety performance.
Prefer to study offline? Printable PDF study book + test engine ($7.50, unofficial). These questions stay free here.