Housing needs reliable incremental SCD merges for eligibility history. Notebook DAG maintenance has become brittle. What should they choose?

Lakeflow Spark Declarative Pipelines are a strong production fit for incremental SCD-style merges when notebook DAG maintenance has become brittle. Housing eligibility history needs reliable, dependency-aware incremental merges with expectations and managed table graphs; declarative pipelines declare that graph instead of stitching fragile notebook task orders by hand. That reduces orchestration drift and keeps SCD Type 1/2-style history updates in a designed refine path under Unity Catalog.

Orchestrating Delta MERGE tasks using Databricks Workflows for incremental SCD merges still stitches together a fragile manually managed task order: hand-ordered merge tasks and manual job graphs make eligibility history loads hard to reproduce and keep brittle. Configuring Auto Loader for incremental eligibility ingestion to maintain SCD history handles incremental file ingestion rather than declarative SCD merges, pushing history logic outside a managed declarative pipeline and leaving history updates brittle and hard to govern. Implementing Structured Streaming foreachBatch upserts for incremental SCD merges reinvents SCD merge logic in custom streaming code instead of using a maintained declarative pipeline, adding operational risk to eligibility history.

Exam caveat: notebooks remain useful for exploring SCD edge cases; the production incremental merge itself should not depend on orchestrated Delta MERGE tasks in Workflows once requirements stabilize. Operational check: run an incremental eligibility change set through the declarative pipeline, verify SCD history rows and current flags, confirm expectations on natural keys, and compare operator runbooks to the pipeline graph rather than to Delta MERGE task order in Workflows.