A model validation team is reviewing an AI scoring model and finds…

Question

A model validation team is reviewing an AI scoring model and finds that the model developer used the full dataset to select features before splitting into training and test sets. What is the primary problem with this approach?

Accepted Answer

It creates data leakage, producing artificially inflated test set performance metrics that do not reflect true out-of-sample performance — Here's the deal — b is correct because using the full dataset (including the test set) to select features means the test set has indirectly influenced model design. This is data leakage: the test set is no longer truly held-out, and the resulting performance metrics are optimistically biased.

Answer

It results in the model having too many features, increasing computational costs

Answer

It violates the model's intended use documentation, which specified that fewer than 20 features be used

Answer

It prevents the model from being retrained on new data without repeating the entire feature selection process

A model validation team is reviewing an AI scoring model and finds that the model developer used the full dataset to select features before splitting into training and test sets. What is the primary problem with this approach?

Short Explanation and Infographic

Full Explanation