An engineering team merges a county resident-services dataset with a city resident-services dataset and profiles the combined table using Glue DataBrew before modeling. The profile surfaces columns with high null rates and several duplicate resident records. What should the team do next?
Select an answer to reveal the explanation.
Short Explanation
A profile report is like a home inspection: it tells you exactly what's wrong, but the inspection itself doesn't fix the wiring. Once DataBrew flags high nulls and duplicate residents, the next move is to actually apply its cleanup recipes and quality rules, not just admire the findings. Skipping straight to training just hands the model a mess and hopes it sorts itself out.
Full Explanation
Profiling is a diagnostic step; DataBrew's transformation recipes and data-quality rules are the remediation step, and both belong in the pipeline before feature engineering starts on merged data. Handling nulls (imputation, targeted removal) and deduplicating resident records directly addresses what the profile found, and doing it before feature engineering prevents downstream features from being computed on corrupted rows. Relying on regularization to compensate for missing and duplicate data confuses a training-time technique for overfitting control with a data-quality problem; regularization doesn't know a duplicate resident record exists, so duplicates simply get double weight in the loss. Discarding the smaller dataset outright throws away legitimate county records because of size, not because of any actual quality signal tied to that source. Repeating the same profiling job without changing the underlying data is a no-op — a profile reflects the data as it exists, and nothing about re-running it changes the data. Scope caveat: cleanup recipes should be reviewed for whether nulls are meaningful (e.g., 'no service requested') before blanket imputation. Operational check: re-run the DataBrew profile after cleanup and confirm null and duplicate rates have measurably dropped.