A court clerk's office wants to compare a new document-classification model variant against the current production model on live traffic before fully switching over. Which approach lets them make that comparison safely?
Select an answer to reveal the explanation.
Short Explanation
Comparing two models is a lot like a taste test; you don't swap the whole menu on faith, you serve both and see which one wins. Running the new variant alongside the current one on live traffic, whether shadowed or split, gives clerks a real, side-by-side comparison before anything gets promoted. That beats a blind swap-and-hope-for-the-best cutover.
Full Explanation
Running an A/B split or a shadow comparison exposes both model variants to real, live document traffic and lets the team measure classification agreement, accuracy, and edge-case behavior before committing to the new variant, directly answering whether it's actually better under production conditions. A full immediate cutover with rollback-on-complaint treats clerks as the detection mechanism, meaning real cases get misclassified before anyone notices, and by then the damage, a wrongly filed document, may already be done. Testing in a local notebook against a handful of samples doesn't reflect the diversity and volume of real docket traffic, so it can hide problems that appear at scale or with unusual document formats. Retraining the current model instead of comparing a distinct variant sidesteps the actual question, since the team specifically wants to evaluate whether the new variant outperforms the existing one, not produce a third option. Scope caveat: a live comparison still needs a clear promotion criterion decided in advance, agreement rate, precision on a specific document class, or the comparison produces data without a decision. A concrete operational check: confirm the comparison logs both variants' classifications against the same document IDs so disagreement cases can be reviewed directly.