Which Databricks feature enables monitoring model prediction drift and data quality in production by storing prediction logs in a Delta table and comparing to a baseline?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Lakehouse Monitoring is Databricks' native drift detection tool — point it at your predictions Delta table, give it a baseline, and it automatically computes statistical drift metrics and generates monitoring dashboards.
Full explanation below image
Full Explanation
Databricks Lakehouse Monitoring (formerly Databricks Model Monitoring) provides: 1) Automated profiling — computes distribution statistics (mean, std, quantiles, null rates) for every column in a Delta table on a schedule. 2) Drift detection — compares production distribution to a baseline (training data or previous time window) using statistical tests (Jensen-Shannon divergence, Wasserstein distance, KS-test). 3) Data quality — monitors null rates, schema changes, value range violations. 4) ML-specific metrics — monitors prediction distribution, label distribution (if ground truth available), and model accuracy metrics. 5) Dashboard — generates SQL-queryable metric tables and Databricks SQL dashboards automatically. Setup: lm.create_monitor(table_name='catalog.schema.predictions', baseline_table_name='catalog.schema.train_baseline', slicing_exprs=['region', 'product_category']). Serving access logs are raw request/response logs, not drift analysis. MLflow Registry has no drift monitoring. DESCRIBE HISTORY tracks table versions, not statistical distributions.