A Spark structured streaming job aggregates loan-tracking scan events (an object moving between branches) into five-minute tumbling windows with a ten-minute watermark. A scan event with an event-time timestamp that is fifteen minutes older than the current watermark then arrives. What happens to that event?
Select an answer to reveal the explanation.
Short Explanation
A watermark is a promise with a limit: Spark will wait up to ten minutes for stragglers, but once an event is older than that, its window has already closed for good. An event fifteen minutes late blows past that promise, so Spark just drops it rather than reopening a finalized window.
Full Explanation
Once a Spark structured streaming query's watermark advances past a window's closing boundary by more than the configured threshold, that window's state is finalized and discarded to keep the state store bounded. An event whose event-time is fifteen minutes behind the current watermark, against a ten-minute watermark threshold, arrives after its window has already been closed and cleaned up, so Spark silently drops that event from the aggregation rather than reopening already-finalized state.
Spark does not automatically buffer an overly late event into a different, still-open window; a late event is only reconciled into its own correct window if it arrives within the watermark threshold, not redirected elsewhere. It also does not restart the query or fail with an exception over a single stale event — dropping late data past the watermark is the normal, expected behavior of the mechanism, not an error condition. Nothing in structured streaming's default watermarking behavior writes dropped-late records to a separate error table automatically; that would require custom logic added by the engineer.
A caveat: the watermark threshold must match the source's real latency profile — too short drops legitimate stragglers, too long delays finalization and grows state. Operationally, monitor the streaming query's dropped-late-event metrics (visible in the Spark UI's streaming query statistics) to confirm how often events are actually arriving past the configured threshold.