A city open-data portal runs a RAPIDS and Dask nightly pipeline and wants reproducible results when the GPU worker pool changes. Which guard best protects reproducibility?
Select an answer to reveal the explanation.
Short Explanation
Think of the declared GPU worker count like the recipe card on a bakery line—if tonight’s oven count drifts, stop the batch instead of hoping the bread comes out the same. Failing the job when workers don’t match the config keeps RAPIDS+Dask civic pipelines honest. Silent resizes are how “same notebook” turns into different citywide numbers.
Full Explanation
Reproducible municipal RAPIDS and Dask pipelines treat declared GPU worker count as part of the experiment contract, not as a soft preference. When an open-data portal’s nightly job starts against a different worker pool than the cluster configuration records, partition boundaries, shuffle topology, and reduction order can change even though the notebook text is unchanged. That resource drift is enough to alter floating-point aggregation paths and civic KPI tables that leadership treats as “the same pipeline.” The strongest guard is to fail the job before processing begins if GPU worker count does not match the declared configuration, forcing operators to reconcile capacity instead of silently shipping divergent citywide outputs.
Ignoring worker count and assuming every Dask schedule yields identical civic results fails conceptually because distributed execution is not schedule-invariant for all reductions and joins; reproducibility requires controlling the resources that shape partitions. Disabling all Dask partitioning so only one CPU core runs the transforms abandons GPU acceleration and Dask’s purpose, trading a configuration guard for a brittle single-core bottleneck that does not address multi-worker reproducibility at all. Deleting the cluster config so operators can resize mid-run removes the very baseline against which drift is detected and invites non-repeatable mid-job topology changes.
Exam caveat: the NCA-ADS workflow-automation stance is fail-fast alignment of declared versus actual distributed resources for reproducibility—not “make it single-threaded” and not “trust every schedule.” Operational check: before the open-data ETL stage, assert current GPU worker count equals the config manifest; on mismatch, exit non-zero, page the platform on-call, and do not write KPI parquet until the pool matches the declared municipal cluster recipe.