Why is file compaction important in a data lakehouse using Apache Iceberg?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Streaming or frequent small-batch writes create many small files that hurt scan performance; compaction (rewriting) merges these into larger optimally sized files; reducing object storage request overhead and improving query speed.
Full explanation below image
Full Explanation
Streaming or frequent small-batch writes create many small files that hurt scan performance; compaction (rewriting) merges these into larger optimally sized files; reducing object storage request overhead and improving query speed. The incorrect options ("To convert Parquet files to Avro for streaming compatibility", "To encrypt data files before storing them in object storage", "To promote tables from the Bronze layer to the Silver layer automatically") are distractors that don't fully capture the concept described.