Analysts need to prototype permit transforms quickly, then promote stable logic into production pipelines. How should notebooks be used?

Permit analytics teams should use notebooks as the prototyping surface—explore transforms on sample permit data quickly—then promote stable logic into declarative pipelines for production. That separation keeps experimentation fast while production gains graph-based, schedulable, testable pipeline definitions instead of scheduled notebook pipelines or versioned pipeline snapshots that drift and lack a single source of truth. Civic data engineering expects an explicit promotion path from sandbox notebooks to production pipelines with owned identities and schedules. Promoting stable logic to production pipelines on all-purpose clusters keeps municipal workloads tied to costly interactive sessions and lacks autoscaled governance, which is unnecessary when promotion to declarative pipelines is the policy. Promoting stable logic to scheduled notebook pipelines for production creates fragile operations, unclear ownership, and hard-to-test municipal releases because code drifts and there is no single declarative definition. Promoting stable logic to versioned pipeline snapshots for production abandons governance, lineage, and durable curated permit tables because snapshots are static copies, not managed pipeline definitions. Exam caveat: notebooks are for development and prototyping; production municipal workloads belong in declarative pipelines after logic stabilizes—not promoted to production pipelines on all-purpose clusters, promoted to scheduled notebook pipelines, or preserved as versioned pipeline snapshots that someone must remember to run. Operational check: prototype a permit transform in a notebook against sample data, extract the stable SQL or DataFrame logic into a Lakeflow Spark Declarative Pipeline, compare notebook versus pipeline outputs on a holdout day, and confirm production does not depend on pipelines on all-purpose clusters, scheduled notebook pipelines, or versioned pipeline snapshots while restricting production writers to the pipeline service identity.