A county social-services case analytics team must test whether a property-tax assistance model generalizes beyond the training period. What evaluation design is most honest?
Select an answer to reveal the explanation.
Short Explanation
You’ll get a more honest read by setting aside a later fiscal quarter and checking how the model handles genuinely later cases. That time-based holdout shows you more than grading the model on work it has already seen.
Full Explanation
To test whether the property-tax assistance model generalizes beyond its training period, hold out a later fiscal quarter and evaluate performance on that temporal split. You train on earlier periods and score genuinely later cases, mirroring how the model will face future applicants and revealing changes in eligibility patterns, documentation norms, and caseload mix over fiscal time. Reusing the identical training rows as the only test set measures how well the model fits or memorizes those rows, not how it generalizes. Randomly mixing records from the later quarter into training lets future-period information leak into model fitting, overstating performance. Reporting training accuracy alone and skipping any holdout also cannot show whether the model works outside the fitted sample. Exam caveat: one later quarter is better than no holdout, but it may be atypical because of seasonal filing spikes or policy changes; document the split dates, and consider rolling temporal validation for stronger evidence. Practical check: freeze features as of each applicant’s decision time, train on quarters before the holdout, score the later fiscal quarter with the production metric, compare subgroup performance across assistance categories, and block promotion if temporal holdout performance collapses relative to in-period scores.