In the Databricks Feature Store, what is the purpose of specifying a timestamp_lookup_key in a FeatureLookup?
Select an answer to reveal the explanation.
Short Explanation
timestamp_lookup_key is what makes the Feature Store time-aware — it tells the join engine to fetch the feature value that existed at the exact moment of each event, preventing future data from leaking into past predictions.
Full Explanation
FeatureLookup(table_name='catalog.ml.user_features', lookup_key='user_id', timestamp_lookup_key='event_timestamp', feature_names=['spend_7d', 'clicks_30d']) performs a point-in-time join: for each row in the spine DataFrame, it retrieves the feature values from user_features that were valid at event_timestamp. This prevents data leakage: if a user's spend_7d changes over time, point-in-time lookup returns the value at the exact event time, not the current (post-event) value. Without timestamp_lookup_key, the join would return the most recent feature values regardless of when the event occurred — introducing future information. The Feature Store's point-in-time join uses the timestamp_keys column in the feature table (set during create_table). This is critical for any time-series or event-based ML problems (fraud detection, churn, click prediction).