A library volume receives new MARC file drops that should be ingested with schema inference as files arrive. What should they use inside declarative pipelines?

When a library volume receives new MARC file drops that should be ingested with schema inference as files arrive, use Auto Loader inside declarative pipelines. Auto Loader watches the volume path, discovers new files, and can infer schema as MARC or related bibliographic drops land—supporting continuous file arrival without manual wiki schemas or annual landing gates. That is the Databricks pattern for incremental file discovery into the lakehouse for library systems. COPY INTO watching the volume path for new MARC drops with schema inference misses the mark for declarative pipelines: COPY INTO is an incremental command that must be re-executed and does not provide continuous file discovery as a streaming source inside a declarative pipeline. Lakeflow Connect watching the volume path for new MARC drops with schema inference is the wrong ingestion pattern: Lakeflow Connect uses managed connectors to ingest from external applications and databases, not to discover and infer schema from file drops landing in a volume. Delta Sharing watching the volume path for new MARC drops with schema inference confuses sharing with ingestion: Delta Sharing securely shares existing data with recipients and does not watch a volume for arriving files or perform schema inference on landing. Exam caveat: Auto Loader provides file discovery and schema inference or evolution options; you still choose a checkpoint location and decide how strict schema evolution should be for MARC variants and vendor quirks. Operational check: point Auto Loader at the library volume path in a declarative pipeline, drop a sample MARC file, confirm it is picked up and inferred or landed, drop a second file and verify incremental ingest, review schema inference logs, and only then promote to production grants for catalog consumers.