An e-commerce recommendation model has been live for several months. Live traffic patterns have shifted, and offline holdout scores no longer match what operators see in production dashboards. Under which condition should the team schedule a retrain of the production model?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: production models don't get a lifetime warranty. Think of your model like a map of last year's city—if streets move, the map misleads you. When live scores on new traffic start slipping versus your baseline, that's your signal to retrain. Pay close attention here, because the exam trap is treating retrain as "never," "once a year," or "only if the goal changes." Goals can change, sure—but drift happens even when the goal stays put. Imagine your boss walks in after Black Friday and conversion predictions are toast: you don't wait until next January. You retrain on recent data (with proper validation), redeploy carefully, and keep monitoring. Got it? Sweet. Remember: retrain when performance on new unseen data degrades—not on superstition or a wall calendar.
Full explanation below image
Full Explanation
In production machine learning, a trained model encodes statistical relationships present in historical training data. Those relationships are not guaranteed to remain stable. Customer behavior, seasonality, product catalogs, fraud tactics, sensor calibration, and labeling practices can all shift over time. When the joint distribution of features and labels moves, predictive performance on new live data often declines even if the original offline test set still looks fine. That gap is exactly why operators watch live metrics and schedule retraining when quality on fresh, previously unseen production data degrades relative to agreed baselines.
Retraining is therefore an operational response to evidence, not a philosophical commitment to permanent models. The correct trigger is measurable degradation—accuracy, precision/recall, calibration error, ranking metrics, business KPIs tied to model scores, or statistically significant data and concept drift detectors. Teams then refresh training corpora with recent representative data, re-run validation and fairness checks, compare candidate models against the current champion, and promote only if gains justify the risk.
Never retraining after release is a common misconception that confuses software immutability with statistical validity. A container image may be immutable; the world feeding it is not. Retraining only when leadership rewrites the objective is also incomplete: objective changes do require new problem formulation, labels, and often architecture, but ordinary drift forces retrain even when the KPI definition is unchanged. Fixed annual schedules ignore both early degradation and unnecessary work when the model is still healthy; time-based refresh can be a safety net, but it should complement—not replace—performance- and drift-driven triggers.
Best practice pairs continuous monitoring (prediction quality, input distributions, latency, and feedback loops) with a documented retrain playbook: data selection windows, labeling SLAs, champion/challenger evaluation, rollback criteria, and audit trails. Memory aid: static code, shifting world—retrain when the world beats the model on new data.