Which PySpark construct is used to compute a rolling 7-day sum of transactions per user as a feature?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Rolling aggregations over time windows are exactly what PySpark Window functions are designed for — partition by entity, order by time, and define the frame as the past 7 days.
Full explanation below image
Full Explanation
PySpark Window functions enable computing aggregations over a sliding window without collapsing rows (unlike groupBy.agg). Pattern: from pyspark.sql import Window; from pyspark.sql import functions as F; w = Window.partitionBy('user_id').orderBy('date').rangeBetween(-6, 0) (for 7-day inclusive window using date integers) or rowsBetween(-6, 0) (for 7 preceding rows); df.withColumn('sum_7d', F.sum('amount').over(w)). rangeBetween uses logical range (date values), rowsBetween uses physical rows. The result retains all original rows with the rolling feature added. groupBy.agg collapses to one row per user_id — losing the temporal dimension. pivot restructures the table rather than computing rolling stats. Pandas UDF with rolling is possible (via applyInPandas) but requires sorting per group and is slower than native Window functions.