A 311 resident-services team's retrieval-augmented assistant answers questions from a knowledge base that grows and changes every week. Six months after launch, more answers cite outdated or missing policy text even though the generation model itself hasn't changed. What should the team do to catch this kind of drift going forward?
Select an answer to reveal the explanation.
Short Explanation
Picture a RAG system as a librarian pointing you to a shelf: it doesn't matter how well you read once you're there if the librarian starts pointing at the wrong shelf. Watching only how polished the answers sound hides the fact that retrieval can quietly go stale as the knowledge base changes underneath it. You want eyes on both halves — is retrieval still finding the current material, and is generation still saying it well?
Full Explanation
A RAG pipeline has two independent failure surfaces: the retrieval step, which finds candidate source chunks, and the generation step, which writes an answer from them. As a knowledge base grows and old policy pages get superseded, the retrieval index can start surfacing stale chunks well before the generation model ever changes, so a monitoring setup needs a metric that answers whether the right, current source got retrieved — not just whether the answer read fluently. Watching generation scores alone fails conceptually because a confident, well-written answer can still be built on an outdated source; fluency and correctness are not the same signal. Retraining the FM on a schedule misdiagnoses the layer: the model didn't get worse at writing, the index got stale, so the fix belongs in retrieval and refresh cadence, not model weights. Adding more documents without adding a quality signal on what gets retrieved can even dilute results further if near-duplicate or superseded pages compete with current ones. Scope note: retrieval monitoring needs a refresh cadence tied to how often source documents actually change. Operational check: run a small labeled query set periodically and confirm the retrieved chunks still match known-current source documents.