How can Delta Lake's time travel feature help prevent training-serving skew in ML pipelines?
Select an answer to reveal the explanation.
Short Explanation
Time travel is like a DVR for your data — if your labels were generated on March 1st, you can rewind the feature table to exactly that moment so your training data matches reality, not today's updated values.
Full Explanation
Training-serving skew occurs when features used to generate labels differ from features used at serving time. Delta Lake time travel (df = spark.read.format('delta').option('timestampAsOf', '2024-03-01').load('/path')) allows you to reconstruct the exact state of a table at any historical point. In practice: if a customer's order_count feature is used to generate churn labels on Jan 1, you need the Jan 1 value of order_count in training, not the current value (which has changed). Delta's transaction log (stored in _delta_log/) enables querying any historical version using either timestamp or version number. The Feature Store's point-in-time lookup feature automates this pattern. Auto-retraining, caching inference results, and encryption are all real capabilities of Databricks but unrelated to the time travel concept.