Data ingestion

BDPC · 30 questions

• Department systems generate continuous events that must land reliably in the analytical platform. What is the primary role of a data ingestion system?
• Traffic sensors should publish once while fraud, maintenance, and dashboards each consume independently. What role do messaging systems play?
• A city event bus keeps ordered records in distributed logs that apps publish to and subscribe from by topic. Which technology pattern matches that design?
• Fare gates emit tap events onto the city's event bus as riders enter. In Kafka terms, what role are the gates playing?
• A Spark job and a separate audit service both read the same fare-tap topic without blocking each other. What Kafka role do those readers play?
• Servers in the city Kafka cluster store topic partitions and serve produce and fetch requests. What are those servers called?
• Operations wants AVL locations, fare taps, and 311 requests kept as separate streams on the event bus. How should those streams be organized?
• A high-volume 311 topic must scale across workers while preserving order for each citizen request key. What Kafka mechanism provides that balance?
• Emergency alerts should be published once while many departmental systems process them asynchronously. Which topic mechanism does that describe?
• After a bad deploy, analysts need to re-read last Tuesday's sensor topic. What Kafka attribute makes that possible within the configured window?
• Stadium concert weekends drive a surge of transit and safety events. How do Kafka-style systems typically scale to absorb that growth?
• A smart-waste pilot will generate millions of small IoT messages each hour. Which Kafka attribute is most directly relevant?
• Real-time alerting workers should share the 311 topic load, while a separate audit pipeline must see every message on its own. How do consumer groups enable that?
• A parser bug corrupted last week's derived parking metrics. After fixing the code, what Kafka operational benefit lets the team rebuild cleanly?
• Architects propose keeping seven years of raw AVL solely inside Kafka with multi-day retention settings unchanged. What correction should leadership make?
• A city 311 bus starts shipping optional JSON fields for newly added service categories while older mobile apps still consume the topic. What ingestion design best tolerates that schema evolution?
• A parking-sensor topic uses at-least-once delivery, so a garage analytics consumer may see the same occupancy event more than once. What consumer practice matches that delivery model?
• Transit events for an entire metro region are spread across many Kafka partitions for throughput. What ordering guarantee should planners expect?
• Flood-alert producers keep publishing while a slow emergency dashboard consumer falls behind. What ingestion design principle should the city apply?
• A municipal payments topic carries fee and refund events that many microservices can see on the shared bus. What security basic should the platform enforce?
• Operations wants traffic-signal telemetry available if the primary datacenter fails. Which ingestion continuity approach fits that goal?
• Streetlight sensors speak constrained field protocols, while the analytics platform expects Kafka producers. Which pattern best bridges that gap?
• Permit-office staff update a transactional database all day, and analysts need those changes as near-real-time events on Kafka. Which ingestion pattern fits?
• A malformed sanitation-route event repeatedly crashes a parser in the main Kafka consumer group. What handling approach keeps the healthy stream moving?
• Live traffic analytics start showing stale congestion maps even though producers are publishing. Which operational metric should on-call engineers watch first on the bus?
• Dispatchers need second-level visibility into ambulance-bay status, but the current feed is a nightly FTP of CSV dumps. Why is that pattern a poor fit?
• One city Kafka topic must refresh a low-latency NoSQL hot store and also land long-term copies in HDFS. Which architecture idea applies?
• A grants team asks for exactly-once end-to-end processing of every subsidy event across producers, Kafka, and multiple sinks. What tradeoff should architects communicate?
• Agencies keep inventing incompatible topic names for similar water-quality events, and producers cannot find the right bus. What governance practice helps?
• A county moves from self-hosted Kafka to a managed cloud messaging service for 311 events. What stays conceptually true?