A courts pilot separates training data (past ruling summaries) from evaluation data held out for testing. Why hold data out?

A courts pilot that separates training data (past ruling summaries) from evaluation data held out for testing is applying a fundamental model-development control. Held-out test sets help check whether the municipal model generalizes beyond examples it trained on, reducing the risk that reported accuracy only reflects memorization of the training summaries. Separation supports honest comparison before any courtroom or clerk workflow relies on the tool. Separating train and test sets to select the municipal model settings using performance on that test set misunderstands the control, since settings should be chosen with a separate validation split — the held-out test set must stay locked for a final independent check, or it loses independence. Separating train and test sets does not help estimate the municipal model accuracy by reusing training examples for testing; re-testing on training data only rewards memorization and defeats the independent check on new examples. Separating train and test sets does not help monitor the municipal model training loss across successive training epochs; training loss is tracked on the training data during optimization, not with a held-out set for generalization. Exam caveat: leakage can still occur if near-duplicate rulings appear in both splits or if preprocessing uses full-corpus statistics; stewards should check split hygiene, not only the existence of a test folder. Operational check: document the split method and date cutoff for the courts pilot, freeze the test set against training use, and report evaluation metrics only from that held-out slice before expansion.