A notebook appends a nightly batch of newly digitized-object metadata to an existing Delta table in a Lakehouse. After the upstream extraction process starts including a new provenance field, the append fails with an error indicating the incoming data's schema does not match the target table. What is happening?
Select an answer to reveal the explanation.
Short Explanation
A Delta table has a fixed shape it expects every new batch to match, kind of like a form with a set number of boxes to fill in. When the upstream source suddenly hands over a batch with one extra box, an append in strict mode says no rather than guessing where the extra field belongs. That refusal is a safety feature working exactly as designed, not a malfunction.
Full Explanation
Delta Lake enforces schema on write by default, comparing an incoming batch's columns against the target table's existing definition and rejecting the write outright when they diverge, which is precisely what happens here once the upstream extraction adds a provenance field the table was never defined with. This behavior exists to stop silent, uncontrolled schema drift from corrupting a table that other processes and reports depend on having a stable, known shape. A memory exhaustion error would typically show resource-related symptoms during processing rather than a message specifically about mismatched schemas. Running out of storage space would produce a capacity-related failure at the storage layer, not a structural comparison between the incoming and existing column sets. A lost connection mid-write would surface as a network or session failure, not a targeted complaint about the data's shape. The concrete resolution path, once the mismatch is confirmed as intentional and permanent rather than an upstream mistake, is to add the new column to the target table's schema deliberately before the next append, rather than forcing the write through without first deciding whether every downstream consumer of that table is prepared for the new field.