In the Databricks Feature Store, what is the purpose of specifying a timestamp_lookup_key in a FeatureLookup?
Select an answer to reveal the explanation.
Short Explanation and Infographic
timestamp_lookup_key is what makes the Feature Store time-aware — it tells the join engine to fetch the feature value that existed at the exact moment of each event, preventing future data from leaking into past predictions.
Full explanation below image
Full Explanation
FeatureLookup(table_name='catalog.ml.user_features', lookup_key='user_id', timestamp_lookup_key='event_timestamp', feature_names=['spend_7d', 'clicks_30d']) performs a point-in-time join: for each row in the spine DataFrame, it retrieves the feature values from user_features that were valid at event_timestamp. This prevents data leakage: if a user's spend_7d changes over time, point-in-time lookup returns the value at the exact event time, not the current (post-event) value. Without timestamp_lookup_key, the join would return the most recent feature values regardless of when the event occurred — introducing future information. The Feature Store's point-in-time join uses the timestamp_keys column in the feature table (set during create_table). This is critical for any time-series or event-based ML problems (fraud detection, churn, click prediction).