A product team ships two candidate ranking models to live traffic and measures which one better meets business KPIs with real users. What practice does this describe in AI product development?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Okay, let's dive in. Offline metrics look great—then you ship and users bounce. That's why A/B testing exists: version A vs version B in the wild, same product surface, different model (or UX), measure clicks, conversion, latency—whatever your KPI is. Think of it like two menu boards at a coffee shop; you see which line moves money. Exam trap: confusing this with “train accuracy vs test accuracy,” which is still one model in the lab. Another trap is “train on two datasets” or “fairness check only.” Related ideas, different jobs. When your boss says “which ranker wins on real traffic?” you're in A/B land. Land the takeaway: A/B = compare live variants with users, not just score one checkpoint offline.
Full explanation below image
Full Explanation
A/B testing (online experimentation) compares two or more variants—models, features, or interfaces—by exposing populations of users under controlled conditions and measuring outcomes on predefined metrics. In AI systems this often means routing a fraction of traffic to a champion model and the rest to a challenger, then estimating uplift in engagement, revenue, retention, or task success while monitoring guardrail metrics such as latency and error rates.
The method matters because offline holdout scores can disagree with production behavior due to distribution shift, feedback loops, position bias, and multi-objective tradeoffs. Sound experiments need clear hypotheses, adequate sample size, randomization or other unbiased assignment, primary and secondary metrics, and safeguards against peeking or multiple-testing errors. Results inform ship/no-ship decisions and continuous improvement in MLOps.
Distractors reflect common mix-ups. Comparing training and test performance is offline evaluation of a single model, not a live two-variant experiment. Training on two datasets concerns data composition, not user-facing comparative testing. Fairness evaluation may run offline or online but is a different objective: A/B testing is the experimental framework for comparing variants, not the definition of fairness analysis. Underlying principle: production truth beats offline proxies when user behavior and feedback loops matter. Best practice pre-registers metrics, powers experiments adequately, and separates guardrails from the primary KPI so latency or safety regressions are not ignored. Memory aid: A/B means two experiences, real users, compare KPIs. Offline split metrics mean one model and a labeled holdout. When a question stresses two versions and which performs better in use, choose A/B testing.