A transit agency streams live bus-sensor data into a Kinesis data stream and needs to apply lightweight unit conversion and deduplication to each record before it lands in Amazon S3, with sub-second processing per record. Which approach best fits this requirement?
Select an answer to reveal the explanation.
Short Explanation
Think of it like an assembly-line inspector checking each item as it passes, instead of sorting the whole day's output into bins after closing time. A lightweight function that fires on every streaming record keeps the fix inline and fast, which is exactly what sub-second unit conversion and deduplication call for.
Full Explanation
Mechanism: a Lambda function attached to the stream is invoked per record or micro-batch, applying stateless transforms like unit conversion and deduplication before the data is written to S3, and it scales automatically with stream throughput while keeping per-record latency low. Why the wrong options fail by concept: an EMR/Spark job on a fixed hourly schedule is a correct tool for distributed batch processing, but its cadence introduces exactly the latency the sub-second requirement rules out, it's the right idea at the wrong layer. A warehouse-plus-scheduled-SQL approach requires loading data first and querying it afterward, adding load time and cost that a streaming inline fix avoids entirely. Glue DataBrew's daily cleanup is a visual batch-profiling tool aimed at periodic dataset cleanup, not per-record streaming transforms, so it can't meet the near-real-time requirement. Scope caveat: at very high throughput, Lambda concurrency and timeout limits become a real constraint, and the stream's shard/partition count needs to be sized so the function keeps pace. Operational check: watch the stream's iterator age metric in CloudWatch after deployment to confirm the function is processing records as fast as they arrive rather than falling behind.
Prefer to study offline? Printable PDF study book + test engine ($7.50, unofficial). These questions stay free here.