Meridian's loyalty-fraud model reaches 98% technical accuracy, yet finance reports that total fraud losses on the frequent-flyer program have not measurably decreased since deployment. What does CPMAI's approach to Model Evaluation say the project team should do?
Select an answer to reveal the explanation.
Short Explanation
High accuracy and real business impact aren't the same thing. If fraud losses haven't budged, something between 'model flags it' and 'losses go down' is broken — maybe alerts aren't acted on fast enough — and that gap is exactly what evaluation should chase down.
Full Explanation
CPMAI insists evaluation criteria trace back to business outcomes, not just technical metrics, and this scenario is a textbook mismatch: 98% technical accuracy but no measurable reduction in actual fraud losses means the model's outputs aren't translating into the intended business result. The right response is to investigate the gap — perhaps flagged cases aren't being acted on quickly enough, perhaps the model catches fraud after the loyalty points have already been redeemed, or perhaps the accuracy metric is measuring the wrong thing (e.g., catching low-value fraud while missing high-value account takeovers) — and adjust the evaluation criteria or the surrounding workflow accordingly. Declaring the project a full success based on the technical number alone repeats the exact mistake CPMAI evaluation is designed to catch: metric success without business success. Simply raising the accuracy target and re-running the identical evaluation doesn't address why accuracy isn't converting to loss reduction — it treats the symptom, not the diagnosis. Abandoning the initiative entirely is a drastic overcorrection; the pattern described (accurate model, absent business impact) usually points to a process or deployment gap, not a fundamentally unsuitable use case. The exam point: technical performance and business KPI achievement are evaluated together, and a gap between them is itself an evaluation finding worth investigating.