A county's feature-engineering pipeline must join a property-tax table with tens of millions of rows against a permitting table of similar size, and a single-node transform tool times out on the join. Which approach is best suited to complete this join reliably?
Select an answer to reveal the explanation.
Short Explanation
Imagine one person trying to sort a warehouse full of boxes alone versus a whole crew splitting the load across aisles. A join across tens of millions of rows on each side needs a crew, which is exactly what a distributed Spark job on a cluster gives you.
Full Explanation
Mechanism: running the join on Amazon EMR with Spark distributes both tables across a cluster of nodes, partitioning the data and executing the shuffle and join operation in parallel, which scales past the memory and compute ceiling that a single machine hits when tens of millions of rows on each side need to be matched. Why the wrong options fail by concept: Data Wrangler's built-in join operator runs on a single interactive instance sized for exploratory data prep, it hits the same single-node resource limits that already caused the timeout, so it doesn't solve the underlying scale problem. A Lambda function is designed for short, lightweight invocations with bounded memory and execution time, holding two massive tables in memory as CSV and merging them there runs straight into those limits. Skipping the join and only profiling with DataBrew abandons the actual requirement, profiling alone never produces the joined feature set the downstream model needs, it just describes the data that was never joined. Scope caveat: even on EMR, join performance depends heavily on how the join keys are distributed, a small number of keys with disproportionately many matching rows can bottleneck a subset of nodes regardless of cluster size. Operational check: review the Spark UI's stage timeline for the job and confirm shuffle time and partition sizes are reasonably balanced across executors rather than concentrated on a few.