A data scientist fits a StandardScaler on the entire dataset before splitting into train/test sets. What problem does this cause?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Fitting the scaler on all data before splitting is like letting a student see the exam answers before taking the test — performance looks great, but it's meaningless in the real world.
Full explanation below image
Full Explanation
Data leakage occurs when information from the test set influences the training process, producing evaluation metrics that don't reflect real-world performance. StandardScaler computes mean and standard deviation from the data it's fit on. If fit on the full dataset (train + test), the scaler's statistics (mean/std) incorporate test set information. When the model is evaluated on the test set, the test features were transformed using their own statistics — the scaler effectively 'knows' the test data. The correct approach: split first (train_df, test_df = train_test_split(df)), then fit_transform on train_df only, then transform (not fit_transform) on test_df using the train-fitted scaler. In Spark ML, using Pipeline() and calling pipeline.fit(train_df) automatically handles this correctly — fit() is only called on training data.