A transit authority's ridership data lake is queried heavily through Amazon Athena, where most queries filter on a handful of columns out of dozens available and occasionally use complex predicate pushdown on nested fields. The team is choosing between ORC and Parquet as the storage format. Which consideration should drive the choice for this Athena-centric workload?
Select an answer to reveal the explanation.
Short Explanation
Think of ORC and Parquet as two well-built filing systems — both columnar, both good at letting Athena skip straight to the columns a query actually needs. Neither one is a magic bullet or a hard requirement; the right pick comes down to testing which one your specific queries and existing Glue tooling handle better for predicate pushdown and compression. Reaching for row-based CSV instead throws away the columnar advantage this selective-query workload depends on.
Full Explanation
Both ORC and Parquet are columnar storage formats that Athena supports natively, and both enable column pruning and predicate pushdown, letting a selective query skip both irrelevant columns and, within supported cases, irrelevant row groups or stripes based on the query's filter conditions — the real differentiator for a given workload is often compression efficiency and how well a specific engine version optimizes pushdown for each format, which is why testing against representative queries matters more than picking by reputation. Treating the two as identical and choosing by naming convention ignores that real differences in compression ratio and pushdown behavior do exist and can measurably affect scan cost and query latency. Recommending CSV for a selective-query workload gets the format tradeoff backwards: a row-based format forces Athena to read every column for a matching row even when the query only needs a few, which is the opposite of what this workload wants. Claiming Athena cannot read ORC files at all is factually incorrect — Athena supports ORC as a native input format alongside Parquet. Scope note: if the data also feeds a non-Athena consumer with weaker ORC support, that consumer's compatibility should factor into the format decision too. Operational check: run the same representative query against equivalent ORC and Parquet copies of a sample dataset and compare Athena's reported data-scanned metric before committing.