Ingest pipelines write large files into HDFS for durable analytics storage. What conceptual write behavior should operators expect?
Select an answer to reveal the explanation.
Short Explanation
When the pipeline lands a big file, HDFS does not just tip its hat. It places blocks on DataNodes and replicates them so one dead disk is not a disaster.
Full Explanation
When ingest pipelines write large files into HDFS for durable analytics storage, operators should expect data to be written to DataNodes with replication as part of durable write semantics. Durability for civic datasets is designed in through distributed block copies and acknowledgment of replicas, not left as an optional afterthought after midnight USB copies. Writes do not skip DataNodes and keep only a pointer inside the NameNode checklist—the NameNode tracks metadata, not the bulk bytes of meter or sensor files. HDFS does not omit replication because durability is optional for municipal data; replication is a fundamental durability and availability mechanism. Every write need not be manually copied by staff to three USB sticks after midnight when the filesystem replication policy is correctly configured and monitored. Exam caveat: the write pipeline is client to DataNodes plus the replica pipeline, while the NameNode records the block map for later reads. Operational check: set and verify replication factor for civic datasets, monitor under-replicated blocks after ingest spikes, alert when writes appear successful in metadata but replicas lag behind policy, and include replica health in the morning platform checklist.