A 311 query profile shows a shuffle-heavy join to a small category dimension. What mitigation fits?
Select an answer to reveal the explanation.
Short Explanation
Tiny category lookup, giant 311 fact—broadcast the little one so you don't ship the whole warehouse across town for every join.
Full Explanation
When a 311 query profile shows a shuffle-heavy join to a small category dimension, the fitting mitigation is to broadcast the small category dimension so the large fact side avoids a full shuffle join. Broadcasting a small civic lookup (categories, codes, types) is a standard Spark optimization when the dimension fits broadcast thresholds and the profile shows shuffle dominance on that join. Increasing shuffle partitions for the join and tuning adaptive skew handling on the fact side fails because it still shuffles the large fact against a tiny dimension instead of broadcasting the small category lookup. Repartitioning the large fact tables by the join key to rebalance the shuffle exchange stage fails because it still moves the big fact side around rather than broadcasting the small category dimension. Caching the large fact DataFrames in memory and scaling executor memory to absorb shuffle spill fails because it throws memory at a shuffle problem that broadcasting the small dimension would avoid entirely. Exam caveat: DP-750 query-profile remediation pairs small-dimension joins with broadcast; do not confuse that with repartitioning the fact tables, caching the fact DataFrames, or tuning adaptive skew on the fact side instead of broadcasting the small dimension. Operational check: confirm dimension size is broadcast-eligible, apply broadcast (hint or auto-broadcast), re-run the profile, and verify shuffle bytes and stage time drop for the 311 join.
Prefer to study offline? Printable PDF study book + test engine ($7.50, unofficial). These questions stay free here.