A PySpark notebook uses Spark structured streaming to compute a five-minute tumbling window count of oral-history recording-upload events from branch archives, but a network outage occasionally delays some upload-confirmation events by up to twenty minutes. Which technique should the engineer add to the streaming query so those delayed events can still be incorporated into the correct window without the state store growing indefinitely?

Spark structured streaming handles late-arriving events through watermarking: withWatermark on the event-time column tells the engine how far behind the latest seen event-time it should keep accepting updates for a given window before finalizing and discarding that window's state. Setting the threshold to at least twenty minutes here means an upload-confirmation delayed by up to twenty minutes still gets aggregated into its correct tumbling window, while the engine can safely drop older window state, keeping the state store bounded.

Increasing executor memory addresses a resource-pressure symptom, not the underlying correctness problem: without a watermark, the state store keeps every open window indefinitely regardless of how much memory is available, and eventually still exhausts it. Switching to a static DataFrame join abandons streaming aggregation altogether — a static join has no notion of time-bounded windows or late data at all. Repartitioning by file extension changes how work is distributed across executors for parallelism; it has no bearing on when a window is considered complete or how late data is reconciled.

A caveat: setting the watermark threshold too high delays how quickly results become final, trading latency for completeness. Operationally, check the query's streaming metrics for the watermark value in each micro-batch to confirm it is advancing and dropping only the intended late window state.