A PySpark job joins a very large fact table of digitised-object records against a small lookup table of the network's dozen branch codes and names. The job spends most of its time shuffling the huge fact table across the cluster just to match it with a handful of lookup rows. Which technique is most directly aimed at eliminating that unnecessary shuffle?
Select an answer to reveal the explanation.
Short Explanation
Shuffling the giant table across the network just to match it with a dozen tiny lookup rows is like mailing an entire warehouse's inventory to every store just so each store can check twelve product names. It makes far more sense to mail the twelve-row list to every store and let each one check its own shelf locally. That's a broadcast join: the small table gets copied to every executor so the big one never has to move.
Full Explanation
A broadcast join sends a copy of the small table to every executor, letting each executor perform the join against its own local slice of the large table without any shuffle of the large dataset across the network — since the branch lookup table is tiny and the fact table is huge, this eliminates exactly the expensive shuffle described, and it's the standard technique for a large-table-to-small-table join pattern. Repartitioning the large table into a single partition would eliminate parallelism entirely, forcing all the data through one task and making the job dramatically slower, not faster — it doesn't address the shuffle problem, it just moves it to a single bottleneck. Caching the large fact table keeps it in memory across multiple uses, which helps if the table is read repeatedly, but caching alone doesn't change how a join is executed — Spark can still shuffle a cached table during a join unless the join strategy itself is changed. Adding an ORDER BY on both tables sorts output rows; it doesn't influence which join strategy Spark chooses or prevent a shuffle. A caveat: broadcasting only makes sense when the smaller table can comfortably fit in each executor's memory — broadcasting a table that's too large can cause out-of-memory errors instead of a speedup. A concrete check: inspect the job's execution plan for a BroadcastHashJoin versus a SortMergeJoin, and compare shuffle read/write metrics before and after forcing the broadcast.