During a custom distributed training loop using Spark, how can you count the number of rows that failed validation across all executors?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Accumulators are Spark's tally counters for distributed tasks — workers can only add to them, the driver can only read them. Perfect for counting errors, skipped rows, or logging events across the cluster.
Full explanation below image
Full Explanation
Spark Accumulators: acc = spark.sparkContext.accumulator(0) (integer, initialized to 0). In an executor function (map, flatMap, UDF), you can add: acc.add(1). On the driver, you read: acc.value. Properties: write-only from executors (executors cannot read the accumulator value). read-only from driver (driver cannot add during task execution). Fault tolerance: if a task is re-executed (due to failure), the accumulator may be double-counted — use only for approximate monitoring, not exact correctness logic. Python global variables are NOT synchronized across executors — they exist independently in each Python worker process and changes are invisible to the driver and other executors. Writing to Delta from each executor is possible but has high overhead for simple counting. Broadcast variables are read-only from executors — you cannot write to them from executor code.