A notebook that runs PySpark cleanup on OCR output from digitised oral-history transcripts is assigned a fixed-size Spark pool sized for the largest transcript batch the archive ever processes, even though most nightly batches are far smaller. Utilization metrics show the pool sitting mostly idle on typical nights, while capacity costs stay constantly high. What change addresses this most directly?
Select an answer to reveal the explanation.
Short Explanation
Renting a full moving truck every single day because you occasionally move a couch is expensive most days you're really just carrying a few boxes. Autoscaling is like renting the size truck the day actually needs — a small van most nights, the big truck only when the big batch shows up. Letting the Spark pool scale between a minimum and maximum instead of sitting fixed at peak size matches capacity to the real, varying workload.
Full Explanation
A fixed-size Spark pool provisions the same number of nodes regardless of the day's actual data volume, so sizing it for the rare largest batch means every smaller, more typical night pays for capacity it doesn't use — exactly the idle-utilization, high-cost pattern described. Configuring the pool to autoscale between a minimum and maximum node count lets Fabric add nodes when a job's workload genuinely needs them and release nodes when it doesn't, right-sizing compute to each night's real transcript volume instead of provisioning for the worst case every time. Switching from PySpark to T-SQL would require restructuring the whole cleanup logic around a different transformation engine and isn't a free substitution — meaningful PySpark-based OCR cleanup logic doesn't translate directly into T-SQL, so this doesn't address the actual sizing problem. A larger fixed pool would only make the idle-capacity problem worse, not better, since it raises the baseline cost further without addressing the mismatch between provisioned and needed capacity. Moving the notebook to a different workspace changes organizational and capacity-allocation boundaries but doesn't change how many nodes are provisioned for a given run. A caveat: autoscaling has to warm up nodes as it scales out, so extremely short jobs may not benefit as much as long-running ones with real workload variance. A concrete check: compare average node utilization and total capacity consumption across a week of typical nights before and after enabling autoscaling.