When orchestrating a multi-step ML pipeline (data prep → feature engineering → training → evaluation → deployment) using Databricks Jobs, which feature enables conditional execution based on the previous step's output?
Select an answer to reveal the explanation.
Short Explanation
Databricks Jobs has built-in DAG task dependencies — you can wire tasks together with conditions like 'only run deployment if evaluation succeeded', creating true conditional ML pipelines without external orchestrators.
Full Explanation
Databricks Jobs (multi-task jobs) support: 1) Task dependencies — specify which tasks must complete before a given task starts, creating a DAG (Directed Acyclic Graph). 2) Run-if conditions — tasks can be configured to run only if dependencies succeeded, always run, or run only if dependencies failed (for error handling). 3) Task values — tasks can pass output values (e.g., validation accuracy) to downstream tasks using dbutils.jobs.taskValues.set('accuracy', 0.92). Downstream tasks read these: dbutils.jobs.taskValues.get(taskKey='evaluation', key='accuracy'). 4) This enables: conditional deployment (only deploy if accuracy > 0.90), alert-on-failure tasks, and retry logic. The combination of task values + run-if conditions creates fully conditional pipelines without external orchestrators. Airflow can be used but is not required. MLflow callbacks don't trigger job steps.