Transit AVL enrich notebooks OOM during wide joins. Beyond hoping it stops, what should engineers tune?

Transit AVL enrich notebooks that OOM during wide joins need engineers to tune partitioning strategy and cluster sizing—and related Spark join settings—to relieve memory pressure, not merely hope the failure stops. Wide joins that assemble large AVL payloads per partition overwhelm executor memory; repartitioning keys, increasing cluster resources appropriately, and adjusting join strategy (for example broadcast where safe) are the substantive levers for municipal fleet enrichments. Caching intermediate AVL DataFrames in memory (and related persistence settings) fails because it loads additional objects into an already strained JVM heap during wide joins, increasing executor memory pressure instead of relieving it. Raising the broadcast threshold for large AVL dimensions (and related join hints) fails because broadcasting a large table ships it to every executor and can exhaust executor memory, making a wide-join OOM worse rather than better. Applying Z-ordering and compaction to Delta AVL tables (and related optimize settings) fails because it improves file retrieval and data skipping but does not reduce executor memory pressure or change how wide joins hold payloads in memory. Exam caveat: DP-750 troubleshooting expects partition, sizing, and join-setting remedies for notebook OOM—not caching in memory (and related persistence settings) that deepens memory pressure, raised broadcast threshold for large AVL dimensions (and related join hints), or Z-ordering and compaction (and related optimize settings) that leave join memory untouched. Operational check: reproduce the OOM with Spark UI memory metrics, adjust partitions, cluster size, and join settings, then confirm the AVL enrich notebook completes without executor loss under representative morning volumes.