A transit authority retrains its ridership-demand model periodically and needs to track and compare metrics across the retrained candidates so the team can decide which one to promote. Which practice best supports this comparison?
Select an answer to reveal the explanation.
Short Explanation
Trying to remember which retrained model performed better without writing anything down is like trying to recall last month's grocery prices from memory — you might get the general shape right, but you won't trust the details. Logging each run's metrics in an experiment-tracking tool integrated with SageMaker gives the team an actual side-by-side comparison to promote from, instead of a vague impression.
Full Explanation
Experiment tracking captures metrics, parameters, and artifacts for each training run in a structured, queryable way, so when multiple retrained candidates exist, the team can compare them directly on the metrics that matter (accuracy, error rate, or a business metric like forecast deviation) rather than relying on memory or ad hoc notes. Leaving metrics scattered across local log files on individual training instances makes systematic comparison painful and fragile — those logs can be lost when an instance terminates, and there's no unified view across runs. Informal recollection of how a model 'felt' during a spot-check isn't a comparison at all; it has no reproducibility and no way to justify the promotion decision later if questioned. Retraining only a single candidate per cycle sidesteps the comparison problem by removing the alternative entirely, which forfeits the chance to catch a retraining run that regressed compared to what a different hyperparameter choice might have produced. Scope caveat: tracked experiments should log enough metadata (data version, hyperparameters) to make a later comparison meaningful, not just the final metric value. Operational check: before promoting a candidate, pull up its tracked run alongside the currently deployed model's run and confirm the new candidate's metrics genuinely improve on it.