A permitting department's application dataset contains near-identical records for the same permit, created when residents accidentally submitted a form multiple times. Left uncorrected, these near-duplicates could bias a permit-approval-time model toward whatever pattern those repeated records happen to show. What should happen during data preparation?
Select an answer to reveal the explanation.
Short Explanation
Three copies of the same complaint letter don't mean the issue happened three times, they mean someone hit submit three times. Deduplicating those near-identical permit records keeps one real event from getting triple the influence it deserves.
Full Explanation
Mechanism: identifying near-duplicate records, whether through exact matching or fuzzy matching on key fields, and removing or consolidating them ensures each genuine permit application contributes proportionately to the training data, preventing accidental resubmissions from skewing the learned pattern toward whatever characteristics those repeated records happen to share. Why the wrong options fail by concept: assuming more rows always improve performance ignores that these aren't independent new observations, they're the same event counted multiple times, which inflates that event's influence on the model rather than adding genuine signal. Converting resubmission counts into a feature treats an artifact of the intake process as if it were meaningful information about the permit itself, when in fact it reflects user behavior at form-submission time, not anything about the application's actual approval timeline. Increasing training epochs addresses how long the model trains, it does nothing to correct the underlying data imbalance caused by duplicated records, the model will simply overfit to the duplicated pattern for longer. Scope caveat: near-duplicate detection needs a sensible matching threshold, records that are similar but represent genuinely distinct permits, like two different residents applying for the same standard renovation type, shouldn't be collapsed together. Operational check: after deduplication, compare the record count per unique permit to confirm no single application is still represented by more than its legitimate submissions.