A feature engineering pipeline computes 50 derived features from raw data. To avoid recomputing these features on every model training run, what is the best practice in Databricks?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Cache vanishes when the cluster restarts. Pickling pandas doesn't scale. The right move is writing features to a Delta or Feature Store table — persistent, versioned, and shareable across teams.
Full explanation below image
Full Explanation
df.cache() and df.persist() store data in cluster memory/disk but are cleared when the cluster terminates or is restarted, making them unsuitable for persisting features across sessions. Pickling pandas DataFrames to DBFS doesn't handle schema evolution, versioning, or concurrent access. The best practice is: 1) Run the feature engineering pipeline once (or on a schedule). 2) Write the result as a Delta table (df.write.format('delta').save(path) or .saveAsTable('schema.table')). 3) For training, read directly from the Delta table — reads are fast and the table can be versioned. Using the Databricks Feature Store adds metadata, discoverability, and point-in-time lookup capabilities. This pattern also enables feature reuse across multiple models.