In a champion/challenger model comparison using Databricks, what is the most rigorous way to determine if the challenger outperforms the champion?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Offline metrics tell you which model looks better on paper. An A/B test with significance testing tells you which model actually produces better business outcomes in the real world — and whether the difference is real or just noise.
Full explanation below image
Full Explanation
Rigorous champion/challenger evaluation requires: 1) Online A/B test — split live traffic (e.g., champion 90%, challenger 10%) using Databricks Model Serving traffic splitting. 2) Business metric measurement — track actual business outcomes (conversions, click rates, fraud caught, revenue) for each model's predictions. 3) Statistical significance testing — after collecting sufficient samples, use a hypothesis test (t-test, chi-square, or bootstrap) to determine if the observed metric difference is statistically significant (p < 0.05). Minimum sample size via power analysis. 4) Practical significance — ensure the effect size is meaningful for the business, not just statistically significant. Why not offline-only evaluation: train-test set may not represent current production data distribution. Training metrics always favor newer models regardless of generalization. Most-recent-date selection ignores model quality. Offline AUC comparison is necessary but not sufficient — online A/B testing is the gold standard.