A team reports excellent accuracy on the training set, yet the same model performs poorly on new, previously unseen examples. What is the most likely diagnosis and a sensible first-line remedy?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Hopefully you answered overfitting—here's why. When training accuracy looks amazing but fresh data tanks, the model basically memorized the homework and failed the real quiz. Think of it like this: a student who only memorizes last year's exact answers aces the practice packet and bombs a new test. Same pattern. First levers: get more (and more varied) training examples, and turn on regularization—weight decay, dropout, early stopping, simpler capacity, data augmentation—whatever fits your stack. Exam trap number one: calling this underfitting. Underfit models are weak on training data too. Trap number two: "add more features" as a default fix—that can make memorization worse. And no, you don't declare victory when unseen data is bad. Pay close attention: track train versus validation curves every run. If they diverge hard, you're overfitting. Got it? Sweet—fix the gap, don't just chase training score.
Full explanation below image
Full Explanation
Generalization is the ability of a learned model to perform well on data drawn from the same process as training but not seen during fitting. When training metrics are strong while performance on a held-out validation or test set is weak, the model has likely overfit: it captured noise, spurious correlations, or idiosyncrasies of the training sample instead of stable structure. High capacity models, small datasets, label noise, and insufficient regularization all increase this risk. Learning curves typically show training loss continuing to fall while validation loss rises or plateaus at a worse level.
The correct diagnosis is therefore overfitting, and practical remedies include increasing the amount or diversity of training data, applying regularization techniques such as L2 weight decay, dropout, early stopping, or constrained architectures, and using data augmentation when inputs support it. Cross-validation, simpler models, feature selection that removes pure noise dimensions, and better label quality also help. Ensemble methods and careful hyperparameter tuning via validation—not test—sets are common complementary tactics.
The distractor that proposes adding still more features misreads the problem. Extra irrelevant features can expand hypothesis space and worsen overfitting or invoke curse-of-dimensionality effects unless accompanied by more data or stronger regularization. The underfitting option confuses two failure modes: underfitting yields mediocre training performance because the model class or optimization cannot capture the signal; simplifying further would not address a large positive train–test gap. Claiming the model already generalizes well contradicts the stated poor results on unseen data.
Operationally, always maintain a clean evaluation split (or nested validation) that is not used for fitting or endless manual peeking. Monitor both training and validation metrics, inspect error slices for distribution shift (which can mimic overfitting symptoms), and only promote models based on held-out performance plus production constraints. On certification exams, map “great train, bad new data” immediately to overfitting plus data/regularization remedies, and reserve underfitting language for weak performance on both sides of the split.