A RAG system ingests documents from multiple sources: an internal wiki, email threads, and meeting transcripts. The system frequently retrieves multiple versions of the same information (e.g., an email summarizing a meeting that is also in the transcript). This duplication causes Claude to give over-weighted responses emphasizing duplicated points. What deduplication strategy should the architect implement?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — a is correct because deduplication requires a two-layer approach: pre-indexing near-duplicate detection (MinHash/LSH) removes nearly identical documents before they enter the index, reducing index bloat; MMR at retrieval time handles cases where near-duplicates pass pre-indexing dedup by adding diversity to the retrieved set, penalizing documents that add little new information to the already-selected set. Together these prevent duplicate information from both entering the index and appearing in the retrieved context.
Full explanation below image
Full Explanation
A is correct because deduplication requires a two-layer approach: pre-indexing near-duplicate detection (MinHash/LSH) removes nearly identical documents before they enter the index, reducing index bloat; MMR at retrieval time handles cases where near-duplicates pass pre-indexing dedup by adding diversity to the retrieved set, penalizing documents that add little new information to the already-selected set. Together these prevent duplicate information from both entering the index and appearing in the retrieved context. B is wrong because exact string matching misses near-duplicates with minor formatting differences, different headers, or paraphrased content — an email summary and a meeting transcript can cover identical information without sharing exact strings. C is wrong because discarding entire source types loses information unique to those sources — emails may contain decisions not in transcripts; context from multiple sources is a feature, not a problem. D is wrong because post-processing Claude's response does not prevent the over-weighting from happening during generation — Claude has already over-weighted the duplicated information when forming its response.