Open-data needs to find an expensive shuffle before gold citation publish. Which tooling path should they use?

Open-data teams hunting an expensive shuffle before gold citation publish should use DAG, Spark UI, and query profile views to locate the costly shuffle stage. Those tools expose stage boundaries, shuffle read/write sizes, and physical-plan operators so engineers can see which join or aggregation dominates publish cost for municipal citation datasets. Relying only on cluster event logs and driver stdout to locate the costly shuffle stage fails because cluster lifecycle output reveals nothing about stage-level shuffle read/write or physical-plan operators. Table history and time travel queries to locate the costly shuffle stage fail because version audit trails do not identify which operator shuffles or whether broadcast versus sort-merge choices are appropriate. Unity Catalog lineage graphs and sharing logs to locate the costly shuffle stage fail because lineage and sharing telemetry show dataset dependencies and consumption, not execution plans or shuffle bytes transferred between executors. Exam caveat: DP-750 performance investigation centers on DAG, Spark UI, and query profile—not cluster event logs and driver stdout, table history and time travel, or lineage graphs and sharing logs to locate the shuffle stage. Operational check: open the citation publish job’s DAG and Spark UI, use the query profile to pinpoint the shuffle-heavy stage, then apply a targeted fix (broadcast, partition redesign, or filter pushdown) and re-profile before gold publish.