A branch's ticketing point-of-sale system occasionally re-sends the same transaction record after a network retry, so a Lakehouse table built from a PySpark notebook ends up with exact duplicate rows sharing the same transaction ID. Which PySpark approach removes those duplicates before the data is written downstream?

dropDuplicates(), optionally scoped to the transaction ID column, is PySpark's direct mechanism for collapsing rows that share the same key, which is exactly the retry-duplicate pattern described; a window function like row_number() partitioned by transaction ID and filtered to the first occurrence achieves the same result when more control over which duplicate to keep is needed. Caching a DataFrame only affects where Spark stores intermediate results for reuse; it has no effect on the actual row content and does nothing to remove duplicates. Repartitioning changes how data is physically distributed across Spark's executors for performance reasons, but it doesn't compare rows against each other or eliminate any of them. Filtering out null transaction IDs addresses a missing-data problem, not a duplicate-data problem — the retried rows in this scenario have a valid, non-null transaction ID that's simply repeated, so a null check wouldn't catch them at all. Before writing the deduplicated data downstream, spot-check a few transaction IDs that were known to retry to confirm exactly one row survived, not zero.