A branch occasionally uploads inter-branch loan-scan events up to two hours late due to intermittent connectivity at remote sites, far longer than what a real-time Spark structured streaming watermark can reasonably tolerate without ballooning state. Which pattern should the team adopt to still capture these very late scans without keeping years of streaming state open?

The standard pattern for reconciling near-real-time streaming aggregation with occasional very-late data is to split the responsibility: the streaming job keeps a modest watermark tuned to typical, expected latency so it can finalize windows and bound its state promptly, while a separate scheduled batch process periodically reprocesses the affected historical time range to incorporate stragglers that arrived well outside the streaming watermark and correct the previously-finalized results. This gives fast, mostly-accurate real-time numbers plus an eventual, fully accurate reconciled figure, without forcing the live pipeline to hold state indefinitely.

Setting the watermark to several days defeats the purpose of watermarking: state for open windows would sit for days, the exact unbounded-growth problem watermarking exists to prevent. Disabling watermarking entirely removes any bound on state growth at all, which is worse, not better, and would eventually exhaust cluster resources. Discarding anything more than ten minutes late and accepting an undercount sacrifices correctness entirely, when a reconciliation pattern can recover it without that trade-off.

A caveat: this only works if downstream consumers can accept a corrected figure arriving after the original real-time number, rather than treating that first value as final. Operationally, schedule the reconciliation batch job to run just after the branch's known worst-case upload delay window and verify its corrected totals against a manually audited sample of late-arriving scans.