After a streaming Spark job writes thousands of small Parquet files to an Iceberg table in watsonx.data; what operation should be run periodically?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Streaming ingestion creates many small files; Iceberg's rewrite_data_files compaction procedure merges them into larger files; improving scan performance by reducing the number of file open/read operations during queries.
Full explanation below image
Full Explanation
Streaming ingestion creates many small files; Iceberg's rewrite_data_files compaction procedure merges them into larger files; improving scan performance by reducing the number of file open/read operations during queries. The incorrect options ("DELETE and re-ingest the table from scratch", "Convert the table from Iceberg to Delta Lake format", "Re-partition the table with a hash partition strategy") are distractors that don't fully capture the concept described.