A global asset manager is building a RAG-based equity research platform that must query a corpus of 40 million SEC filings, broker research reports, and proprietary analyst notes. The system must return grounded answers in under two seconds with citation-level attribution. Which architectural decision is most critical to achieving both latency and attribution requirements at this scale?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of it like a library reference desk — you first check the card catalog (BM25 sparse search) to narrow 40 million books to 500 candidates, then send your sharpest librarian (dense reranker) to pick the best 10. Persisting chunk-level provenance IDs is what lets you put the exact footnote in the answer. That two-stage pattern is the industry-standard solution for high-volume, latency-sensitive, attribution-critical RAG.
Full explanation below image
Full Explanation
Retrieval-Augmented Generation at enterprise scale requires careful separation of concerns between retrieval speed, retrieval precision, and source traceability. At 40 million documents, a single-stage dense vector search across the full corpus would require prohibitively large GPU memory for MIPS-optimized approximate nearest-neighbor (ANN) indexes, and would return results in seconds even on optimized infrastructure like FAISS HNSW or Pinecone — well over the two-second budget once the generation step is added.
The two-stage pipeline solves this elegantly. BM25 sparse retrieval (using an inverted index such as Elasticsearch or OpenSearch) is extremely fast at scale — sub-100ms even for 40 million documents — and has excellent recall for keyword-heavy financial queries involving ticker symbols, exact clause language from filings, or named entity terms. This first pass reduces the candidate set from 40 million to a manageable 200-500 chunks. The dense bi-encoder reranker (such as a fine-tuned sentence-transformer or cross-encoder model) then operates over this small candidate set with high precision, ranking chunks by semantic relevance to the query. Total retrieval latency is well within budget, leaving ample headroom for LLM generation.
The citation requirement makes chunk-level provenance ID persistence non-negotiable. Each text chunk must carry an immutable reference to its source document (filing accession number, broker report identifier, or internal research note ID), page number or section, and ingestion timestamp. When the LLM generates a response, these provenance IDs are threaded through the context and surfaced in the output as footnotes or inline citations. Without this design, the system can only attribute answers to a document at best, not to a specific passage — a material weakness for regulatory and compliance use cases.
Fine-tuning the LLM on the corpus (option C) embeds knowledge parametrically but loses attribution entirely — the model cannot cite where a claim came from — and requires prohibitively expensive re-training whenever new filings are published. Partition-based parallel search (option D) can reduce per-index latency but does not address the reranking precision gap and still requires a merge step that introduces its own latency and cosine-score comparison artifacts across heterogeneous document types.
The FINRA guidance on AI explainability and the SEC's focus on audit trails for AI-assisted research both reinforce the architectural primacy of traceable, citation-level attribution in financial AI systems, making the two-stage pipeline with provenance persistence the correct answer.