While running 5-fold cross-validation, a data scientist notices that the model's accuracy swings wildly between folds — scoring 96% on one fold and only 71% on another, despite the folds being drawn from the same overall dataset. What does this high variance across folds most likely indicate?
Select an answer to reveal the explanation.
Short Explanation
Big swings in accuracy across folds are a classic overfitting signature — here's why. If your model is memorizing quirks of whatever training slice it happens to see, then its performance becomes totally dependent on which random subset it got. Test it on a fold that looks a bit like its training data, and it does great; test it on a fold with slightly different noise, and it falls apart. That's high variance, textbook overfitting. An underfitting model, by contrast, tends to perform consistently mediocre everywhere because it never learned much to begin with — its scores don't swing, they just stay low. So consistent bad scores point to underfitting; wild, inconsistent scores point to overfitting.
Full Explanation
In the bias-variance framework, variance refers to how much a model's predictions change when trained on different samples of data. A model with high variance has effectively learned the idiosyncrasies and noise specific to whatever training fold it saw, rather than the general underlying pattern, so its performance is highly sensitive to exactly which data points ended up in the training versus validation split. This is the textbook signature of overfitting: strong, sometimes near-perfect performance on data resembling what it memorized, and a sharp drop when evaluated on data that differs even slightly, which is precisely the 96% vs 71% swing described. The fix typically involves regularization (L2/dropout/early stopping), more training data, or reducing model capacity. Underfitting is incorrect because an underfit model has high bias, not high variance — it fails to capture the underlying pattern in the data at all, so its errors tend to be systematic and fairly consistent across different folds; you would expect uniformly mediocre scores (e.g., 74%, 73%, 76%, 75%, 72%) rather than wide swings. Data leakage is incorrect as the most likely explanation here because leakage (where information from the label or future data improperly leaks into the features) typically produces uniformly and suspiciously high performance across all folds and even on held-out test data, not high variability between folds — leakage tends to inflate scores consistently rather than cause instability. An overly low learning rate is incorrect because that primarily affects training speed and convergence within a single training run (the model may train too slowly or get stuck before reaching a good minimum), but it does not inherently explain why performance would differ so drastically depending on which fold was used for validation; a slow-converging model trained for a fixed number of epochs would tend to underperform somewhat consistently across folds rather than oscillate wildly. The key underlying principle: consistent-but-poor performance signals bias/underfitting, while inconsistent/highly variable performance across resamples of the same data signals variance/overfitting.