A PySpark notebook processes a large nightly batch of OCR text extracted from scanned oral-history transcripts. One cell calls .collect() to pull the entire distributed dataset into the driver for a quick inspection, and that cell now fails on especially large batches while earlier cells that only build and cache Spark DataFrames succeed. What does this pattern point to?

The .collect() action gathers every partition of a distributed Spark DataFrame back onto the driver node as a local, in-memory Python structure, so its memory cost scales with the full size of the dataset rather than staying spread across the executors the way earlier caching and transformation steps do. On a large enough OCR batch, that concentrated memory demand can exceed what the driver has available, producing a failure specifically at the collect step while prior cells that build or cache the distributed DataFrame keep succeeding, since those operations never require the whole dataset to live on one node. A syntax error would fail immediately regardless of batch size and would not correlate with data volume the way this failure does. A missing import would typically fail the very first time that library is referenced, not intermittently based on how much data is being processed. A missing Lakehouse table would fail at the read step, before the DataFrame the collect call operates on could even be built. As an operational check, replace the diagnostic .collect() with a bounded alternative such as taking a small sample or writing a summary aggregation instead, and confirm the failure disappears once the driver is no longer asked to hold the entire batch.