A transit agency streams live bus-sensor data into a Kinesis data stream and needs to apply lightweight unit conversion and deduplication to each record before it lands in Amazon S3, with sub-second processing per record. Which approach best fits this requirement?
Select an answer to reveal the explanation.
Short Explanation
Think of it like an assembly-line inspector checking each item as it passes, instead of sorting the whole day's output into bins after closing time. A lightweight function that fires on every streaming record keeps the fix inline and fast, which is exactly what sub-second unit conversion and deduplication call for.
Full Explanation
Mechanism: a Lambda function attached to the stream is invoked per record or micro-batch, applying stateless transforms like unit conversion and deduplication before the data is written to S3, and it scales automatically with stream throughput while keeping per-record latency low. Why the wrong options fail by concept: an EMR/Spark job on a fixed hourly schedule is a correct tool for distributed batch processing, but its cadence introduces exactly the latency the sub-second requirement rules out, it's the right idea at the wrong layer. A warehouse-plus-scheduled-SQL approach requires loading data first and querying it afterward, adding load time and cost that a streaming inline fix avoids entirely. Glue DataBrew's daily cleanup is a visual batch-profiling tool aimed at periodic dataset cleanup, not per-record streaming transforms, so it can't meet the near-real-time requirement. Scope caveat: at very high throughput, Lambda concurrency and timeout limits become a real constraint, and the stream's shard/partition count needs to be sized so the function keeps pace. Operational check: watch the stream's iterator age metric in CloudWatch after deployment to confirm the function is processing records as fast as they arrive rather than falling behind.