What is the recommended approach for adding unit tests to a Databricks ML pipeline notebook before promoting to production?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Testing in Databricks notebooks uses standard pytest — either run it in the notebook itself or trigger notebooks via CI/CD with the Databricks REST API. There's no special Databricks testing magic.
Full explanation below image
Full Explanation
Testing approaches for Databricks ML notebooks: 1) Inline pytest — write test functions (def test_feature_engineering():...) in a notebook cell and run: !pytest notebook_name.py. 2) Separate test notebooks — create test_feature_engineering.py notebooks and use dbutils.notebook.run('test_notebook', timeout=300) in CI/CD. 3) Databricks Repos + CI/CD — push notebooks to a Git repo, trigger GitHub Actions or Azure DevOps pipelines that use the Databricks Jobs API to run test notebooks on a cluster. 4) dbutils.notebook.run() in workflows — chain notebooks in a Databricks Job where test notebooks run before promotion. MLflow does NOT auto-run unit tests before registration — tests must be explicitly orchestrated. There is no 'Databricks Test Runner UI'. The standard approach for ML projects: write pytest-compatible tests in Python files, run them in CI/CD before merging to main.