A legal department has deployed a Copilot Studio agent backed by a RAG pipeline over a corpus of 800 complex legal contracts, each averaging 45 pages. Users ask highly specific questions such as 'What is the penalty clause in the Northwind vendor contract for late delivery?' The agent currently uses fixed-size chunking at 512 tokens with 0 overlap, and users frequently report that answers are either incomplete or miss the relevant clause entirely. Which chunking strategy change would most directly improve retrieval accuracy for this specific legal use case?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Fixed-size chunking treats documents like a bread loaf sliced at exactly 512 slices regardless of where the sentences end — you often get half a clause on one slice and the other half on the next. Semantic chunking respects natural boundaries (sentences, paragraphs, clauses), with overlap acting as a safety net at boundaries, so each chunk tells a complete story that retrieval models can actually match.
Full explanation below image
Full Explanation
The root problem is that fixed-size chunking at 512 tokens with no overlap will routinely split legal clauses across chunk boundaries. A penalty clause that spans, say, 600 tokens will be cut in half: the retriever will score each half-chunk lower than a complete clause because neither fragment alone contains all the semantic signal the query expects. The result is missed or incomplete answers.
Option A is correct because semantic chunking (also called sentence-aware or structure-aware chunking) splits on natural language boundaries — sentence endings, paragraph breaks, or in the case of legal documents, numbered clause headings. An overlap of 10–15% (roughly 50–75 tokens on a 512-token target size) ensures that the tail of one chunk and the head of the next share content, eliminating cold splits. This preserves clause integrity and dramatically improves recall for precise fact-extraction queries.
Option B is an improvement in isolation — larger chunks reduce split risk, and metadata filtering is a valid optimization for multi-document corpora — but a 4096-token chunk for a 45-page contract will still span many unrelated clauses, polluting the context window with irrelevant text. The LLM's attention is diluted, and the agent's answer quality degrades even when the right clause is technically present. Metadata filtering is a complementary technique, not a replacement for proper chunking.
Option C moves in the wrong direction. Reducing to 128 tokens creates more granular chunks, which increases the probability that a clause is split into even more fragments. Individual fragments of legal clauses are semantically impoverished — the retriever will struggle to match them to a natural-language query. Very small chunks also increase index size and retrieval latency.
Option D — cross-encoder reranking — is a valid production technique for improving the precision of the retrieved-set, but it operates on chunks that already exist. If the chunks themselves are malformed (mid-clause splits), reranking will promote the least-bad fragment, not the correct complete clause. Fixing the root-cause chunking strategy will yield far more improvement than layering reranking onto broken chunks.
Exam tip: For the AB-620 exam, know the three main chunking strategies: (1) fixed-size, (2) sentence/semantic, and (3) hierarchical. Legal, regulatory, and technical documentation benefits most from semantic chunking. Overlap is always recommended except for documents where chunks are already inherently self-contained (FAQ answer pairs, structured tables).