Scanned city-council minutes enter the RAG corpus with OCR garbling of names and motion text. What should preprocessing do before indexing?
Select an answer to reveal the explanation.
Short Explanation
Garbage-in OCR means garbage-out answers about what council actually voted. Clean the noise and normalize the text before you embed those scanned minutes.
Full Explanation
Scanned city-council minutes entering the RAG corpus with OCR garbling of names and motion text require cleanup and normalization before preprocessing and embedding. Cleaning OCR noise (character fixes, reading-order repair, consistent whitespace/punctuation) so embeddings reflect what council actually recorded is a standard municipal document-prep step before public or staff agents retrieve motions.
Indexing raw OCR unchanged to preserve “authenticity” of errors preserves garbage that pollutes retrieval. Upsampling OCR noise so the retriever learns to ignore all council content actively degrades a critical civic source. Replacing minutes with synthetic poetry avoids OCR by abandoning factual content entirely.
Exam caveat: cleanup must not silently rewrite substantive votes—prefer confidence-based review for low-OCR-quality pages. Operational check: compare retrieval and answer accuracy on a held-out minutes question set before/after OCR cleanup, and spot-check that motion text and names match the corrected source rather than garbled embeddings. Retain original scans alongside cleaned text for legal discovery while indexing only normalized content for RAG. Measure OCR character-error rate on sampled council pages and quarantine batches that exceed the cleanup quality threshold.