A city open-data portal sees near-perfect training accuracy, yet scores drop sharply when the model is applied to newly annexed districts. What does that pattern most clearly indicate?
Select an answer to reveal the explanation.
Short Explanation
Aced the practice quiz but bombed the new neighborhoods? That’s overfitting or shift—the model memorized the old city, not the newly annexed map.
Full Explanation
A city open-data portal that sees near-perfect training accuracy yet sharp drops on newly annexed districts is showing poor generalization—most clearly from overfitting to the historical city distribution or from distribution shift relative to the new geography. High train fit with collapsed performance on unseen districts means the model memorized old patterns (or relied on features unstable under annexation) rather than learning transferable structure. That reading matches the classic train–serve gap municipal ML teams must monitor after boundary changes.
Claiming perfect production generalization with no further monitoring needed contradicts the observed score drop on new districts. Saying validation metrics are illegal to compute on open data is false and irrelevant to the pattern. Claiming GPU training always guarantees identical scores on every geography fails by concept—hardware acceleration does not erase overfitting or spatial shift.
Exam caveat: the drop could be pure overfitting, pure shift, or both; annexed districts may also lack feature coverage (new street IDs, missing historical labels), so diagnose before blaming only model capacity. Operational check: compare feature distributions and missingness between legacy and annexed districts, score a dedicated annexation holdout, try simpler regularization or re-training with representative annexation samples, and add geographic monitoring so open-data model cards flag district-level metric regressions after boundary updates.