A fraud-detection model scored well in offline tests, yet after go-live it misses new fraud patterns that appear in live traffic. What is this situation typically called?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out. Your fraud model looks great in the lab—solid AUC, happy stakeholders—and then real crooks invent new tricks. Suddenly the model shrugs. That's not "you over-optimized the learning rate," and it's not underfitting (it did learn the old patterns). Fairness bias is a different conversation. Here's what you're seeing: data drift or model drift. The world moved; features and fraud tactics no longer match what you trained on. Think of it like a spam filter still hunting last year's phishing lures while scammers already switched scripts. Exam trap: people pick underfitting or bias because those words sound technical. Flag it: strong offline metrics plus failing new live patterns = drift. Monitor, retrain, update features. Trust me on this—one that bites in production all the time.
Full explanation below image
Full Explanation
Production machine learning systems operate in environments that change. Fraudsters adapt attack vectors, customers change payment habits, product catalogs evolve, and seasonal effects shift transaction mixes. When the statistical properties of inputs, or the mapping from inputs to labels, diverge from the distribution seen during training and offline evaluation, model quality declines even if the original test set looked excellent. Practitioners commonly refer to this family of issues as data drift (feature or input distribution shift), concept drift (the relationship between features and the target changes), or more loosely model drift when deployed predictive behavior degrades over time. Detecting and responding to drift is a core MLOps concern.
The correct answer names data drift or model drift as production conditions diverge from the training world. In the fraud scenario, new fraud types mean the historical training labels and patterns no longer fully represent live risk. Offline success only proves the model fit a snapshot of past reality. Without monitoring feature distributions, prediction scores, precision and recall on delayed ground truth, and periodic revalidation, silent failure is likely.
Over-optimization of hyperparameters is a training-time concern about chasing validation noise or building brittle configurations; it does not specifically describe post-deployment emergence of novel attack classes. Underfitting occurs when capacity or representation is insufficient to capture even the training signal, typically yielding poor results before go-live. Model bias usually refers to systematic performance disparities or unfair outcomes across groups; while drift and bias can interact, the vignette centers on new fraud types over time rather than demographic fairness alone.
Best practices include establishing baseline distributions at deployment, alerting on significant shifts, maintaining feedback loops for newly labeled fraud, scheduling or triggering retraining, and designing features that remain informative as tactics evolve. Shadow deployments, canary releases, and human review queues help contain risk while models catch up. A useful exam memory aid: if the model worked on the old world and the world changed, think drift—not simply "train a bigger network." Continuous evaluation is as important as the initial accuracy number stakeholders celebrate at launch.