A retailer stores 80 TB of clickstream and orders in S3 with duplicated Parquet datasets. Analysts need ACID upserts, schema evolution, fine-grained access, and SQL from Athena and Redshift. Engineering wants open formats without a proprietary store. Which data lakehouse design supports governed Iceberg queries with Lake Formation?
Select an answer to reveal the explanation.
Short Explanation
You can avoid update chaos from duplicated Parquet datasets with an open-table format on S3. An Iceberg lakehouse governed by Lake Formation gives you upserts and shared permissions for Athena and Redshift Spectrum.
Full Explanation
Apache Iceberg tables on S3 provide open-format ACID transactions, row-level upserts, snapshot isolation, and schema evolution. Lake Formation supplies centralized fine-grained permissions for Athena and Redshift Spectrum, supporting governed multi-engine SQL without duplicating the datasets. Copying all S3 data nightly into provisioned Redshift tables may support SQL queries, but duplicates 80 TB, reduces open-format flexibility, and does not provide lakehouse upserts or centralized lake permissions. Keeping independent Parquet prefixes and sharing bucket policies supports basic cross-team access, but plain Parquet lacks ACID transactions and schema evolution, and bucket policies offer coarser control than Lake Formation; updates can also create inconsistent copies. Loading raw JSON into DynamoDB and exporting weekly CSV files is operational storage rather than an analytical lakehouse: weekly exports are stale, while CSV lacks transactions and efficient columnar queries. Exam caveat: distinguish Iceberg with Lake Formation for governed upserts and multi-engine access from Redshift Spectrum alone, which can query external tables but does not itself provide this governance and table functionality. Practical check: verify Glue Data Catalog integration, Lake Formation LF-Tags, and the IAM role used by Spectrum to access the Iceberg tables.