A reprocessing job may embed the same ordinance version more than once. How should writes be handled?
Select an answer to reveal the explanation.
Short Explanation
Idempotent upserts are the ‘same ordinance stamp, same slot’ rule—rerun the embed job all you want and you still get one tidy vector row, not a pile of twins.
Full Explanation
When a reprocessing job may embed the same ordinance version more than once, writes must be idempotent. Performing upserts keyed by ordinance version (or an equivalent natural key) so repeated embedding runs update the same vector row prevents duplicate municipal embeddings from polluting RAG similarity and citation sets. Inserting a new duplicate embedding row on every reprocess with no natural key multiplies near-identical vectors, wastes storage, and skews similarity results toward over-represented ordinance text. Skipping writing embeddings entirely when reprocessing known ordinance versions can leave stale vectors after model or chunking changes that required a deliberate re-embed. Overwriting unrelated departments’ vectors whenever one ordinance is reprocessed breaks tenant isolation and corrupts other civic corpora sharing the store. Exam caveat: idempotent upsert addresses duplicate risk from retries and reprocessing; it does not replace dimension checks or soft-delete filters on withdrawn content. Operationally, choose a stable ordinance-version key, use upsert/merge semantics on that key, emit metrics for insert-versus-update counts, and audit row counts before versus after deliberate reprocess runs to confirm no duplicates.