An insurer wants to promote a custom fraud model. Nobody has agreed how precise it must be, and no holdout set exists. What must be in place before production?
Select an answer to reveal the explanation.
Short Explanation
You would not pass a class without a passing score or an exam you have not already seen. Before promoting this fraud model, set an agreed precision bar and use a holdout set the model was not trained on. A score on the training claims can flatter the result.
Full Explanation
Validation criteria for a custom model have to exist before promotion, and they must include a bar the business accepts plus examples the training run did not use. This fraud model has neither an agreed precision bar nor a holdout set, so both are required before production. Promoting now and writing down the bar and holdout set after go-live lets the model affect claims before anyone can say whether it failed. A holdout set alone provides unseen claims but still has no line the model must clear; leaving precision for the fraud team to judge by feel each week makes the decision inconsistent. Scoring a precision target only on the same claims used to train the custom model can look strong while saying nothing about claims the model has not memorized. Exam caveat: the bar should be the one the insurer agrees for this fraud decision, not a generic score copied from another model. Operational check: record the agreed precision bar, confirm the holdout claims were excluded from training, and block promotion until the model is scored against that bar on that set.