A RAG system indexes a 2-million-document corpus. New documents are added daily (approximately 500/day), and existing documents are updated weekly. The current architecture performs a full re-index nightly (4-hour job). As the corpus grows, nightly re-indexing will exceed the 6-hour maintenance window. What indexing architecture handles scale without exceeding the window?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — you need incremental indexing because you only process the delta, not the whole corpus. That keeps the daily update job small and predictable as the corpus grows. Write-ahead logging makes sure you don't miss additions or modifications.
Full Explanation
Incremental indexing is correct because it solves the index maintenance scalability problem directly: by processing only the delta (500 new documents per day plus weekly modifications), the daily index update job is O(delta) rather than O(corpus size). This keeps maintenance time bounded regardless of corpus growth. Write-ahead logging ensures no updates are missed during incremental processing, and a full rebuild is reserved only for major schema changes. Increasing server resources is wrong because vertical scaling at best delays the problem — full re-index time grows with corpus size, and scaling hardware linearly to maintain a 6-hour window is expensive and eventually impossible. Reducing indexing frequency to weekly is wrong because new documents would be invisible to the RAG system for up to 7 days, severely degrading freshness for a corpus with 500 daily additions. Sharding the index and re-indexing one shard per night in rotation is wrong because each document would only be re-indexed once per shard rotation cycle (8 days for 8 shards), losing daily update freshness; updated documents in non-rotating shards would remain stale until their shard is rotated.