An iterative model that revisits the same civic graph many times runs far slower on classic disk-heavy MapReduce than on Spark. Why?

Iterative models that revisit the same civic graph many times run far slower on classic disk-heavy MapReduce than on Spark because Spark’s in-memory, iterative strengths reduce repeated disk I/O that classic MapReduce incurs between steps. Each MapReduce iteration often materializes to disk; Spark can keep working sets hotter in memory across iterations. Claiming MapReduce always keeps every intermediate set fully in GPU memory by default is false. Saying Spark forbids iteration so only MapReduce can loop reverses the teaching contrast. Asserting HDFS cannot store graph edges unless Spark is disabled confuses storage with compute engines. Exam caveat: memory is finite—huge graphs still spill; the advantage is relative I/O reduction, not infinite RAM. Operational check: compare an iterative graph scoring job’s runtime and disk spill metrics on a MapReduce-style pipeline versus Spark with caching enabled on a representative civic graph sample. In production terms for Spark in-memory iteration versus MapReduce, assign an owning team, wire a dashboard that shows the pattern is healthy under peak civic load, and require a staging drill that rehearses failure and recovery before resident impact. Revisit the choice when traffic shape, retention, or integrity needs change, and keep a named rollback owner on the change record. Re-test both the happy path and the failure path for Spark in-memory iteration versus MapReduce after material dependency upgrades so regressions are caught before the next public release.