A snow-route model was trained only on weekday downtown plow logs because those trucks already had sensors. Weekend residential cul-de-sacs never appear in the file. A high F1 on the downtown slice is being celebrated. What should the team flag first?
Select an answer to reveal the explanation.
Short Explanation
Think of practicing only downtown weekdays, then celebrating the score while weekend cul-de-sacs never made the file. That is collection bias: the sample does not represent the routes you must plow. A high downtown F1 does not guarantee the streets you never measured.
Full Explanation
Training only on weekday downtown plow logs because those trucks had sensors creates collection bias: weekend residential cul-de-sacs never appear in the file, so the sample does not represent the routes the snow-route model must score in production. A high F1 on the downtown slice can look excellent while saying almost nothing about streets that were never measured. The first flag should be that the training population is not the population of decisions the city will make.
Calling this a Domain 4 policy essay instead of a data problem misplaces the issue. Collection bias is a fundamentals-of-data and ML-design failure: who was sampled, when, and with what sensors. Policy documents may later govern fairness remediation, but they do not redefine biased sampling as non-data.
Blaming overfitting to the inference server's clock casts cul-de-sacs as a hardware timing issue. The gap is geographic and calendar coverage in the training file, not clock skew on the serving host.
Celebrating downtown F1 as a guarantee of weekend cul-de-sac performance ignores representation. Metrics on an easy, well-instrumented slice can hide failure modes on the routes residents actually need cleared on Saturday mornings.
Exam caveat: high offline metrics are only trustworthy when the evaluation set matches the intended deployment mix. Slice metrics can reassure you about downtown weekdays while the operational risk sits elsewhere.
Operational check: build an evaluation report stratified by district and day type; withhold promotion until weekend residential routes have labeled coverage or an explicit, accepted limitation is documented for operators and leadership.