An engineer needs to join a live stream of building-management HVAC events with a live stream of gallery-occupancy events, computing a rolling correlation between occupancy and temperature drift using custom stateful logic that goes beyond simple filtering or aggregation. Which tool should the engineer reach for?

Spark structured streaming supports stream-to-stream joins and arbitrary stateful transformations expressed in PySpark, including custom logic like a rolling correlation calculation that has no equivalent built-in operator on a simpler canvas. Because the requirement here goes beyond filtering, renaming, or straightforward aggregation into genuinely custom stateful computation across two live streams, a Spark notebook is the tool designed to express that.

The Eventstream canvas's built-in operations — Filter, Manage fields, Group by — cover common shaping and windowed aggregation needs, but they do not expose a mechanism for joining two independent live streams together and running a custom statistical computation like correlation; that exceeds the no-code operator set. A T-SQL view over Warehouse tables only queries data that has already landed at rest; it has no access to two continuously arriving live streams and cannot compute a rolling, streaming correlation. A Dataflow Gen2 with a scheduled refresh is a batch tool that pulls data periodically rather than reacting to events continuously, which is fundamentally mismatched with a rolling, live correlation requirement.

A caveat: choosing Spark here trades the simplicity of the no-code canvas for the operational overhead of managing a running notebook job, including checkpointing and cluster sizing. Operationally, validate the join logic first on a bounded static sample of both event types before promoting the notebook to run continuously against the live streams.