A county open-data portal's RAG search assistant starts returning less relevant document excerpts to residents after a content refresh, even though the underlying foundation model hasn't changed. Which diagnostic path should the team prioritize?
Select an answer to reveal the explanation.
Short Explanation
If the answers got worse right after a content refresh and the model itself never changed, the model probably isn't the suspect; the search step is. Retrieval-accuracy monitoring checks whether the vector search is even finding the right documents before the model ever sees them. Fix what actually changed, not the piece that stayed the same.
Full Explanation
Retrieval-accuracy monitoring evaluates whether the vector search step returns documents that are actually relevant to a given query, isolating the RAG pipeline's retrieval stage from its generation stage, and since the timing lines up with a content refresh rather than a model change, the retrieval index, fresh embeddings, changed chunking, a reindexed vector store, is the far more likely culprit. Re-running generation-quality metrics like ROUGE against a fixed reference set tests how well the model summarizes or phrases given source text, but if the model is being handed the wrong source excerpts in the first place, no generation metric explains why the excerpts themselves are irrelevant. Increasing temperature changes how varied or creative the model's phrasing is; it has nothing to do with which documents get retrieved and can make an already-wrong answer harder to evaluate. Swapping to a larger foundation model adds cost and complexity while leaving the root cause, retrieval, completely untouched. Scope caveat: retrieval monitoring should compare results against the same query set before and after the content refresh, since a genuinely different result mix could also just reflect improved, differently-organized content. A concrete operational check: sample a set of resident queries and confirm whether the top retrieved chunks changed in relevance after the refresh, isolating retrieval from generation.