After dedup, a county-fair dump still has two-word fragments, other-language ads, and highly repetitive spam. What should happen before anyone builds a vector index?
Select an answer to reveal the explanation.
Short Explanation
After dedup, two-word fragments, other-language ads, and repetitive spam still sit there. Apply Data Curator-style length, language, and repetition filters before anyone builds a vector index. PII policy, indexing first, and a Megatron fine-tune do not clean that dump.
Full Explanation
Curation is more than dedup. Length, language, and repetition filters are dataset prep that should run before an index is built. PII policy and consent stay in Domain 5. Do not index the fragments and ads first, and do not treat junk as a pretraining corpus.