A fraud-detection model scores excellently on internal lab benchmarks but performs poorly after go-live. Which cause is most likely?

When a model performs well under laboratory conditions yet fails in production, the most probable root cause is a mismatch between the data used for development and the data encountered live. This phenomenon is variously described as dataset shift, distribution shift, covariate shift, concept drift, or train/serve skew, depending on which factors differ. Lab datasets are often cleaner, historically bounded, or sampled from a population that does not fully represent production traffic. Feature pipelines in notebooks may also diverge subtly from online feature computation, producing different inputs than those seen at training time.

The symptom pattern in the question is decisive: excellent lab benchmarks demonstrate that the model can learn and generalize within the offline distribution. Therefore explanations that primarily blame pure hyperparameter neglect, pure architectural complexity, or pure architectural simplicity are weaker. Hyperparameter issues or capacity problems would typically degrade offline metrics as well, or would require additional evidence such as overfitting diagnostics. Complexity can contribute to fragility, and simplicity can limit capacity, but neither is the most direct explanation for a sharp lab-to-production gap when offline results were already strong.

Addressing data mismatch requires deliberate engineering: representative training and validation data, production-like holdouts, shadow scoring on live traffic before cutover, feature store consistency between training and serving, and continuous monitoring of input distributions, prediction distributions, and labeled outcomes when available. Domain experts should review whether labels and business definitions still match operational reality. Techniques such as importance weighting, periodic retraining, and robust validation across time slices further reduce surprise at launch.

From a process perspective, treat production as the ultimate evaluation environment and design offline tests to approximate it as closely as possible. Document data sources, collection windows, known biases, and pipeline transformations. When incidents occur, start investigations with data comparisons before rewriting architectures. Exam memory aid: lab success plus production failure points first to data mismatch—not automatically to 'too big,' 'too small,' or 'never tuned.' That prioritization guides both certification answers and real incident response, where comparing distributions often reveals the gap faster than another hyperparameter sweep alone would under time pressure in operations.