Architects propose keeping seven years of raw AVL solely inside Kafka with multi-day retention settings unchanged. What correction should leadership make?
Select an answer to reveal the explanation.
Short Explanation
Kafka is the short-term inbox; the lake is the archive basement. Keep days on the bus, years in HDFS—not seven years stuffed in the inbox.
Full Explanation
Keeping seven years of raw AVL solely inside Kafka with multi-day retention settings unchanged is a misuse of the log: leadership should treat Kafka as a durable buffer/log for days-scale retention and land long-term history in HDFS or a data lake. Kafka retention windows do not already guarantee multi-decade lake storage by default. Deleting the data lake because brokers supposedly replace all archival storage confuses hot/warm event transport with cold historical analytics storage. Disabling HDFS and keeping only the most recent five minutes of events forever destroys longitudinal transit analysis. Exam caveat: define explicit tiering—Kafka for recent replay, lake/object storage for multi-year AVL—and monitor that sink jobs stay ahead of retention deletion. Operational check: confirm a lake compaction job lands AVL partitions before Kafka retention expires, spot-check a year-old trip against lake storage, and verify broker retention remains sized for operational replay—not for seven-year primary archive. In production terms for Kafka buffer versus long-term lake storage, assign an owning team, wire a dashboard that shows the pattern is healthy under peak civic load, and require a staging drill that rehearses failure and recovery before resident impact. Revisit the choice when traffic shape, retention, or integrity needs change, and keep a named rollback owner on the change record. Re-test both the happy path and the failure path for Kafka buffer versus long-term lake storage after material dependency upgrades so regressions are caught before the next public release.