A water-utility sensor analytics ETL currently writes intermediate stages to CSV between GPU steps even though schemas are stable. What change most improves pipeline efficiency?
Select an answer to reveal the explanation.
Short Explanation
You’re doing extra work when each GPU stage has to turn the same data into CSV text and parse it again. Land stable intermediate stages in Parquet so the next RAPIDS step can reuse typed, columnar data with less I/O and parsing.
Full Explanation
For a water-utility sensor analytics ETL with stable schemas, landing intermediate stages in Parquet (or another columnar analytics format) instead of repeatedly writing and reading CSV most improves efficiency. Columnar intermediates preserve data types, support compression, and make re-reads cheaper for RAPIDS or Dask stages, avoiding repeated text serialization and parsing between GPU transforms. Adding more CSV dialects and quote-escaping rules fails by concept: it can make text handling more complex and fragile, but does not remove the CSV round-trip cost. Exporting each stage to PDF tables fails because PDFs are meant for document presentation, not efficient interchange for analytics transforms. Serializing each intermediate as newline-delimited JSON keeps records readable, but reloading the full text before every GPU stage still requires text parsing and does not provide the same columnar access benefits. Exam caveat: Parquet works best when schemas are stable enough to commit; incompatible ad hoc columns still require schema discipline or evolution rules. Practical check: replace CSV checkpoints with Parquet in a water-sensor ETL branch, measure stage wall time and bytes written, verify that data types survive into the next RAPIDS step, and retire CSV intermediates for stable stages.