A media company promoted a caption model after a strong offline score. A week later, captions on new shows look worse, and nobody is comparing the live error with that offline number. What should the post-release watch do?
Select an answer to reveal the explanation.
Short Explanation
The score that earned the promotion is a promise, and you still have to see whether live shows keep it. Post-release monitoring should watch whether live caption error stays near the offline result. The trap is rescoring the old holdout, which cannot see the new programs.
Full Explanation
The offline score justified the release, and the live question is whether today's captions still land near that score. Post-release monitoring compares the error on live shows with the offline result and alerts when the live number drifts, which is how a worse week becomes visible instead of anecdotal. The offline number stays the reference recorded at promotion, and the live number is recomputed on captions actually served after release. Watching training loss on the original files stays inside the data the model already fit and treats the offline score as frozen. Restart counts speak to process stability on the host, not to caption error, so a calm process can still emit bad captions. Rescoring the original holdout reproduces a number from shows the team already measured and never looks at the new programs that prompted the concern. Exam caveat: a gap between live and offline error can come from a shift in shows or from a serving bug; the watch detects the gap and does not by itself name the cause. Operational check: confirm a dashboard plots live caption error beside the offline score that was recorded at promotion.