A logistics firm wants to compare a new arrival-time model with the current one on real shipments, with both models seeing the same live traffic mix. Which test does that?
Select an answer to reveal the explanation.
Short Explanation
A fair race puts both runners on the same track at the same time. You want an online comparison that sends the same live shipment mix to both arrival-time models. The trap is scoring last quarter's trips, or a quiet depot that never sees the other model's traffic.
Full Explanation
The firm wants both models on real shipments and on the same mix of live traffic, so the comparison is not an artifact of an older week or of a quieter depot. An online comparison does that by splitting current traffic between the two models under the same conditions and recording both outcomes. The split is what makes a difference in arrival error attributable to the model rather than to a different week or a quieter depot. An offline replay scores completed trips from last quarter and never puts either model on a live shipment, so the mix is historical. A training-loss comparison refits models on stored rows and measures fit, not live arrival error under a shared traffic mix. A canary at one quiet depot sends the new model a different population from the depots still on the current model, so the conditions are not the same. Exam caveat: a single-depot canary can be a cautious rollout later; it is not the same-conditions comparison this stem asks for. Operational check: during the test window, confirm both models receive draws from the same live traffic policy and that outcomes are logged for both.