When orchestrating a multi-step ML pipeline (data prep → feature engineering → training → evaluation → deployment) using Databricks Jobs, which feature enables conditional execution based on the previous step's output?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Databricks Jobs has built-in DAG task dependencies — you can wire tasks together with conditions like 'only run deployment if evaluation succeeded', creating true conditional ML pipelines without external orchestrators.
Full explanation below image
Full Explanation
Databricks Jobs (multi-task jobs) support: 1) Task dependencies — specify which tasks must complete before a given task starts, creating a DAG (Directed Acyclic Graph). 2) Run-if conditions — tasks can be configured to run only if dependencies succeeded, always run, or run only if dependencies failed (for error handling). 3) Task values — tasks can pass output values (e.g., validation accuracy) to downstream tasks using dbutils.jobs.taskValues.set('accuracy', 0.92). Downstream tasks read these: dbutils.jobs.taskValues.get(taskKey='evaluation', key='accuracy'). 4) This enables: conditional deployment (only deploy if accuracy > 0.90), alert-on-failure tasks, and retry logic. The combination of task values + run-if conditions creates fully conditional pipelines without external orchestrators. Airflow can be used but is not required. MLflow callbacks don't trigger job steps.