A parks permits RAG team changes how ordinance PDFs are chunked and needs proof retrieval quality improved. What should they use?

Measuring relevance metrics such as nDCG on a fixed municipal evaluation set before and after the chunking change is how a parks permits RAG team proves retrieval quality improved. Chunking changes alter what the index can return; only a controlled before/after comparison on the same ordinance queries and graded relevance judgments shows whether the new boundaries help clerks and applicants cite the right sections. Fixed eval sets make the comparison reproducible for GenAIOps review and change control.

Measuring relevance metrics such as nDCG on a fixed municipal eval set after the chunking change rollout fails because an after-only measurement has no baseline — without the before-change score on the same fixed set you cannot prove improvement, only report a single number. Measuring relevance metrics such as BLEU on a fixed municipal eval set before and after the chunking change fails because BLEU scores text overlap and fluency, not whether the retrieved chunks are the relevant ones — a fluent answer can still be built on the wrong ordinance section. Measuring relevance metrics such as nDCG on separate municipal eval sets before and after the chunking change fails because using different sets for the “before” and “after” measurements breaks the controlled comparison — a fair before/after test needs the same fixed eval set measured both times, not separate sets that could differ in difficulty or topic mix.

Exam caveat: nDCG evaluates retrieval relevance, not generation style or token cost. When the stem asks for proof after a chunking change, pick evaluation on the same fixed set measured before and after, not an after-only score with no baseline, a BLEU score for fluency, or separate eval sets that break the comparison.

Operational check: freeze a parks-permits query set with graded relevant ordinance chunks, compute baseline nDCG (and precision@k) on the old chunking, recompute after the new chunking with identical retriever settings, and gate promotion on a predefined improvement threshold plus a spot-check of citation correctness.