A city streaming micro-batch must not land duplicate ticket_id values inside the same window. How should uniqueness be enforced?
Select an answer to reveal the explanation.
Short Explanation
Uniqueness on ticket_id is like one claim check per bag—two checks with the same number in one batch means something went wrong at the counter. You shouldn't rely on source exactly-once delivery and checkpoint offsets to enforce ticket_id uniqueness within the micro-batch window, since that only prevents reprocessing, not duplicate business keys. You also shouldn't apply streaming dropDuplicates on ticket_id after the sink write to enforce ticket_id uniqueness within the micro-batch window, since you'll have already landed the duplicates and you can't fix it with a Delta sink table MERGE on ticket_id either since it won't stop two same-key rows in one batch.
Full Explanation
Civic streaming and micro-batch quality gates can enforce uniqueness constraints such as ticket_id within a processing window. A city 311 pipeline that must not land duplicate ticket_id values in the same micro-batch should use pipeline expectations (or equivalent quality rules) to detect duplicates and fail, drop, or quarantine according to policy. Enforcing uniqueness in the quality layer prevents double-counted SLA clocks, duplicate work orders, and conflicting field merges before gold aggregates form. Relying on source exactly-once delivery and checkpoint offsets to enforce ticket_id uniqueness within the micro-batch window fails by concept: exactly-once delivery and checkpoint offsets prevent duplicate processing of the same event, but they do not enforce the uniqueness of the business key ticket_id within the window if the source emits duplicates or if the offset is not the ticket_id. Using Delta sink table MERGE on ticket_id as the merge key to enforce ticket_id uniqueness within the micro-batch window fails by scope: MERGE upserts the micro-batch against the existing target table, but it does not enforce uniqueness among rows inside the same micro-batch, so two rows with the same ticket_id in one window can cause a multiple-source-match error or nondeterministic survival instead of a quality fail, drop, or quarantine. Applying streaming dropDuplicates on ticket_id after the sink write to enforce ticket_id uniqueness within the micro-batch window fails by order: deduplicating after the sink write lets both duplicate ticket_id rows land first, so it does not prevent duplicate facts and cannot fail, drop, or quarantine them before gold aggregates form. Exam caveat: uniqueness within a micro-batch window is not the same as global deduplication across all history—late arriving data and cross-window duplicates may still need separate historic handling with watermarking strategies. Operational check: feed two events with the same ticket_id in one window, confirm the uniqueness expectation fires and duplicates do not both land as independent gold facts, then confirm distinct ticket_ids still publish and metrics show the uniqueness violation clearly in the pipeline UI.
Prefer to study offline? Printable PDF study book + test engine ($7.50, unofficial). These questions stay free here.