After an AI model is live in production, which practice best describes responsible maintenance of that model over time?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Okay, let's dive in. Shipping the model is not the finish line—it's the starting gun. Once it's live, the world keeps moving: customers change behavior, fraudsters invent new tricks, seasons shift demand. Your job is to watch the live scoreboard—accuracy, latency, calibration, business KPIs—and when things slip past the threshold you and the business agreed on, you retrain or refresh with fresh data. Think of it like a car: you don't buy it and never change the oil. Exam trap? Options that sound "safe" like never update or train once. Trust me, that freezes a snapshot of yesterday and calls it forever. Different model per prediction? That's chaos, not maintenance. Land this: monitor continuously, retrain when performance degrades. That's production hygiene.
Full explanation below image
Full Explanation
Model maintenance is the post-deployment phase of the machine learning lifecycle in which teams keep a production model accurate, reliable, and aligned with business objectives as data and environments evolve. Unlike a one-time software compile, statistical models encode patterns from historical training data. When the live input distribution shifts (data drift) or the relationship between inputs and targets changes (concept drift), predictive quality degrades even if application code is unchanged. Responsible maintenance therefore pairs continuous monitoring with a defined response: investigation, retraining, recalibration, or controlled rollback.
The correct practice is to watch production performance metrics on an ongoing basis and retrain or otherwise refresh the model when quality drops below agreed thresholds. Typical signals include prediction accuracy or AUC on delayed labels, residual error, calibration plots, slice-level fairness metrics, data-drift statistics (for example population stability index), and operational metrics such as latency and error rates. Thresholds should be documented in service-level objectives so on-call engineers know when action is mandatory rather than optional.
Freezing weights permanently and never updating confuses "controlled change" with "no change." Change control is healthy; permanent stasis is not, because the world is non-stationary. Swapping a completely different model for every single prediction is not maintenance—it destroys consistency, complicates debugging, and usually increases cost without a sound ensemble design. Training only once at launch assumes stationarity; many failures in fraud, recommendations, demand forecasting, and content moderation appear months after go-live when the original training window no longer represents reality.
Best-practice MLOps for maintenance includes automated monitoring dashboards, alerting, labeled feedback pipelines, versioned datasets and models, shadow or canary deployments for new candidates, and human review for high-risk domains. Retraining should be reproducible: same feature definitions, validated data quality checks, evaluation on held-out or time-based splits, and approval gates before promotion. When labels are delayed, use proxy metrics carefully and recalibrate once ground truth arrives. Remember that "retrain" is not always the first fix—broken upstream features, schema changes, or traffic mix shifts may need engineering remediation first. Still, the exam-level principle is clear: continuous monitoring plus conditional retraining when performance degrades is the hallmark of production model maintenance, not one-shot training or rigid freezes.