An insurer's evaluation traces show that policy-rider questions retrieve no usable rider passage, while deductible and limit questions retrieve the right clause and are answered correctly. What should the architect tune?
Select an answer to reveal the explanation.
Short Explanation
When only one shelf in the library is empty, you repair that shelf. You tune the rider knowledge the failing questions need, and you leave the model and the passing topics alone. Dropping rider questions from the next test only hides the gap.
Full Explanation
Tuning starts from the step the evaluation shows is failing. These traces say rider questions retrieve no usable rider passage, while deductible and limit questions retrieve the right clause and answer correctly, so the model can already use a source when the source is retrieved. The architect tunes the rider knowledge: what is indexed, how it is chunked, and whether the agent is allowed to read it. Retraining the whole model treats one retrieval gap as a reason to change every weight, including the paths that already pass. Rewording the deductible answers spends the change on topics the evaluation already marks correct. Dropping rider questions from the evaluation set makes the report look clean and leaves the missing rider passages unfixed. After the rider knowledge is corrected, the same evaluation should be run again to confirm that step and no other. Exam caveat: a failed answer is not automatically a model defect. The trace has to show which step missed. Operational check: rerun the rider questions after the knowledge change and confirm they now retrieve a rider passage, while a deductible question still retrieves its original clause.