A transit authority streams real-time GPS pings via Amazon Kinesis into a data lake and is deciding whether to write raw JSON directly to S3 or add a conversion step to Parquet before landing. The downstream feature pipeline runs frequent, selective Athena queries filtering on route and time window. What is the most relevant tradeoff in this decision?
Select an answer to reveal the explanation.
Short Explanation
Think of raw JSON like handwritten notes and Parquet like a typed, indexed report — the notes are faster to jot down, but every lookup means reading page by page. Converting to Parquet costs a bit of processing time up front, but it pays that back when Athena only has to touch the route and time-window columns a query actually needs. For frequent, selective queries, that tradeoff favors doing the conversion.
Full Explanation
The real tradeoff here is a one-time ingestion-time cost against a recurring downstream cost: converting streamed JSON to Parquet adds processing latency and complexity at ingestion, but it pays off because Parquet's columnar layout lets Athena's frequent, selective route-and-time-window queries scan only the relevant columns and row groups instead of full JSON records, cutting scanned bytes and query cost on every single query afterward. Framing raw JSON as preferable purely because it avoids an extra processing step ignores that the downstream query pattern is frequent and selective — exactly the pattern where columnar formats pay for themselves quickly. Claiming Athena scans a similar number of bytes regardless of format misstates how Athena scanning works: row-based JSON requires reading full records even for narrow queries, while Parquet's columnar pruning measurably reduces scanned bytes for column-selective queries. Assuming Kinesis locks the pipeline into JSON and that conversion later requires rebuilding ingestion from scratch overstates the difficulty — a downstream Glue or Lambda transformation step can convert already-landed JSON to Parquet without re-architecting the streaming ingestion path itself. Scope note: for infrequent or non-selective queries, the conversion overhead may not be worth it, so the decision should follow actual query patterns rather than a blanket rule. Operational check: compare Athena's reported bytes-scanned for a representative query against equivalent JSON and Parquet copies of a sample before committing to the conversion step.