A Spark structured streaming job aggregates artifact-scan events into five-minute tumbling windows and writes results to a Lakehouse table using append output mode. The team notices that windows only appear in the output table well after they close, and never get corrected once written. Which streaming concept explains this behavior?

In Spark structured streaming, append output mode guarantees that a row, once written to the sink, is never updated or removed afterward. To honor that guarantee for windowed aggregations, Spark withholds a window's aggregated result until the watermark has advanced far enough to confirm no more late data can arrive for that window, at which point the result is finalized and emitted exactly once. That mechanism is precisely why the team sees windows appear only after they close, with no subsequent corrections.

Append mode does not emit raw, unaggregated rows immediately — that describes stream pass-through with no aggregation at all, which is not what the job's windowed tumbling aggregation is doing. Lakehouse tables, built on Delta, fully support streaming writes; that is a standard and common target for Spark structured streaming sinks, so it is not the explanation here. A watermark threshold of zero would not disable output entirely; it would simply mean Spark considers a window closed the instant its end boundary passes, with no tolerance for late data — a different (and much stricter) behavior than what is described.

A caveat: a use case that needs partial, in-progress window totals rather than waiting for finalization should use update output mode instead, where the sink supports it. Operationally, compare the timestamp a window closes against the timestamp its row appears in the Lakehouse table to confirm the delay matches the configured watermark threshold.