A transit authority runs a new ridership-prediction model as a shadow variant, receiving live traffic alongside the current production model but with its predictions withheld from riders, before deciding whether to promote it. Why is this evaluation approach useful before promotion?
Select an answer to reveal the explanation.
Short Explanation
Think of shadowing a new bus driver on the actual route before letting them carry passengers — they experience the real road, real traffic, real surprises, but nobody's at risk if they make a wrong turn. Running a model in shadow mode does the same thing: it sees genuine live traffic while its predictions stay hidden from riders, surfacing real-world issues offline testing might never catch. That's why it's a safer bridge between validation and full production.
Full Explanation
Shadow deployment lets a candidate model process real, live production traffic in parallel with the current production model, with its outputs logged and compared but never actually served to end users, which surfaces behavior differences that only emerge under genuine live data patterns, load, and edge cases that a static offline validation set may not represent. This gives the team evidence about how the new model would actually perform in production before committing riders to its predictions. The option claiming shadow evaluation permanently replaces the need for a held-out validation set overstates its role; offline validation and shadow testing serve complementary purposes, and shadow testing is typically a step after, not a replacement for, standard validation. The option describing automatic promotion after a fixed number of requests misrepresents shadow deployment as an unconditional trigger rather than a data-gathering step that still requires human or automated evaluation of the comparison results before any promotion decision. The option claiming the production model must go offline during shadow evaluation contradicts the entire point of the technique, which is running the new variant alongside the existing production model precisely so live service continues uninterrupted while the comparison happens. Scope note: shadow testing adds infrastructure cost, since two models process the same traffic simultaneously, so it is typically time-boxed. Operational check: compare the shadow model's predictions against the production model's actual outcomes over the shadow period before making a promotion decision.