A school district uses Apache Flink on Amazon Kinesis Data Analytics to compute windowed aggregations (for example, hourly attendance counts per building) from attendance-sensor streams before the results are stored for modeling. Why does performing this aggregation in the streaming layer, rather than as a later batch job against raw stored events, benefit the downstream modeling pipeline?
Select an answer to reveal the explanation.
Short Explanation
Think of it like tallying votes as ballots come in instead of dumping every single ballot into a box and counting them all again later — you get the same trustworthy count, but with a lot less to store and less work downstream. Streaming aggregation in Flink turns a firehose of raw sensor events into compact, hourly summaries before they ever hit storage. That's the real win here: leaner storage and less redundant processing, not some accuracy boost or a free pass on messy data.
Full Explanation
Computing windowed aggregations in the streaming layer lets Flink summarize a high-volume raw event stream into compact records (an hourly count per building, say) before anything lands in storage, which shrinks both the storage footprint and the amount of data any downstream batch job has to scan later, and it keeps the aggregation logic co-located with Flink's native event-time windowing, built specifically to bucket streaming events by time correctly. Claiming the streaming aggregation is inherently more accurate than an equivalent batch computation misattributes the benefit; a correctly implemented batch job against the same raw events can produce an equally accurate aggregate, and the real advantage is efficiency and pipeline simplicity, not correctness. Saying this removes the need to store raw events entirely overreaches — many teams keep raw events for auditability, reprocessing after a bug fix, or training a model that needs event-level rather than aggregated detail, so aggregation typically supplements rather than replaces raw storage. Claiming it eliminates handling late or out-of-order events misunderstands watermarking: Flink's windowing still needs late-data and watermark configuration to decide how long to wait before closing a window, and it doesn't make the ordering problem disappear on its own. Scope note: pick a window size and lateness allowance based on how attendance data actually arrives from the sensors, not a default. Operational check: compare a sample hour's Flink-computed count against a manual tally from raw stored events to confirm the aggregation logic is correct before relying on it for modeling.