The 311 pipeline must standardize addresses in-process on the job cluster. What should the data engineer do?

When the 311 pipeline must standardize addresses in-process on the job cluster, the data engineer should install the required Python geocoding library on the job compute used by that pipeline. Cluster- or job-scoped libraries make the package available to executors during the address-standardization stage so transforms run inside the governed Databricks job rather than in side channels. That is the straightforward compute-library pattern for civic ETL enrichment.

Storing the geocoder as a Unity Catalog row filter on the tickets table misuses row filters (which restrict which rows a principal may see) as a software distribution mechanism; filters do not install Python packages or perform geocoding. Replacing address fields with random hashes avoids the library but destroys the address standardization the pipeline requires and harms downstream routing analytics. Running geocoding only inside Azure Monitor alert action groups places enrichment in an observability/alerting product instead of the data pipeline, which is neither scalable nor appropriate for batch address cleansing.

Exam caveat: libraries belong on the compute that runs the code—job cluster libraries for job workloads, all-purpose cluster libraries for interactive development. Unity Catalog governs data access; it is not a pip/Maven replacement. Prefer workspace/job library declarations over ad-hoc %pip drift when production reproducibility matters, though exam stems that simply say “install on job compute” are satisfied by attaching the library to that job compute.

Operational check: add the geocoding package as a job-cluster library (PyPI/Maven as appropriate), run a 311 sample job, verify workers import the module during address standardization, and pin versions so nightly ticket transforms do not break on silent upstream package changes.