A media company ingests 8,000 click events per second from web clients, must transform JSON to Parquet in near real time with no servers to manage, auto-discover schema for analysts, and support ad-hoc SQL without clusters. Which serverless data processing pipeline meets these streaming and analysis requirements?
Select an answer to reveal the explanation.
Short Explanation
You need to turn a fast stream of clicks into affordable storage you can query without managing servers, and this pipeline fits. Kinesis Data Streams, Lambda, S3, Glue, and Athena provide serverless processing and analysis from JSON to Parquet.
Full Explanation
Kinesis Data Streams handles ingestion at the stated rate, Lambda transforms the incoming JSON into Parquet in S3, Glue discovers and catalogs the schema, and Athena provides ad-hoc SQL without clusters. Together, these services meet the near-real-time, no-server-management, and analyst-discovery requirements. Managed Flink to OpenSearch is a good fit for real-time search and dashboards, but its search-oriented output does not provide the S3 Parquet data lake and Glue catalog needed for the requested SQL analysis. SQS FIFO with Fargate tasks and RDS MySQL can move and query data, but the tasks require scaling and management, while a relational database is a poor fit for sustaining 8,000 analytics writes per second. MSK with EMR Spark Streaming and provisioned Redshift can handle streaming and SQL, but the provisioned brokers, processing cluster, and warehouse add operational work and conflict with the serverless requirement. Exam caveat: choose the Flink-to-OpenSearch design when real-time search and dashboards are the goal; for custom JSON-to-Parquet transformation and SQL over S3, the Kinesis Data Streams and Lambda pipeline is the better match. Before launch, verify Parquet file sizes, Glue table partitions, and Athena workgroup limits.