A RAG system indexes a 2-million-document corpus. New documents are added daily (approximately 500/day), and existing documents are updated weekly. The current architecture performs a full re-index nightly (4-hour job). As the corpus grows, nightly re-indexing will exceed the 6-hour maintenance window. What indexing architecture handles scale without exceeding the window?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — b is correct because incremental indexing is the correct architectural solution to index maintenance scalability: by processing only the delta (500 new + modified documents per day), the daily index update job is O(delta) rather than O(corpus size). This keeps maintenance time bounded regardless of corpus growth.
Full explanation below image
Full Explanation
B is correct because incremental indexing is the correct architectural solution to index maintenance scalability: by processing only the delta (500 new + modified documents per day), the daily index update job is O(delta) rather than O(corpus size). This keeps maintenance time bounded regardless of corpus growth. Write-ahead logging ensures no updates are missed during incremental processing. A is wrong because vertical scaling (more server resources) at best delays the problem — full re-index time grows with corpus size, and scaling hardware linearly to maintain a 6-hour window is expensive and eventually impossible. C is wrong because reducing indexing frequency to weekly means new documents are invisible to the RAG system for up to 7 days, severely degrading freshness for a corpus with 500 daily additions. D is wrong because rotating shard re-indexing means each document is only re-indexed once per shard rotation cycle (8 days for 8 shards), losing daily update freshness; and updated documents in non-rotating shards are stale until their shard is rotated.