A water-utility demand desk is scoring weekly consumption forecasts for a drought contingency pilot. Which evaluation practice best fits the series?
Select an answer to reveal the explanation.
Short Explanation
You’ll get the most useful score by testing the forecast on later weeks that match the planning window, using MAPE or RMSE. That lets you judge how it may perform in practice, rather than relying on shuffled weeks or a visual check of training data.
Full Explanation
For a water-utility demand desk scoring weekly consumption forecasts in a drought contingency pilot, evaluation must mirror how planners use the series. Weekly demand is ordered in time: weather, irrigation rules, conservation messaging, and holiday effects create serial dependence. Scoring with MAPE or RMSE on a chronological holdout that matches the planning window preserves the real gap between the last observed week and the weeks the model must predict, so the error estimates operational risk rather than in-sample fit. Reporting only training-set R-squared after randomly shuffling weeks destroys temporal order and turns forecasting into an i.i.d. regression exercise; high shuffled training R-squared can coexist with poor forward forecasts when regime shifts or seasonal structure matter. Judging the forecast by its fit to the week’s highest-demand day, without reserved test observations, cherry-picks one extreme and ignores average and near-term path error across the full contingency horizon. Comparing forecasts only by visual inspection of training weeks is subjective, non-reproducible, and biased toward memorizing history the model has already seen. Exam caveat: a chronological holdout is not k-fold CV on shuffled rows; for ordered civic series, leakage across time is the primary failure mode. MAPE can also be unstable near zero consumption, so pair it with RMSE when drought curtailment drives values toward zero. Practical check: lock a rolling-origin or expanding-window protocol whose forecast length equals the drought desk’s planning lead time, document the metric suite before scoring, and reject candidates whose training metrics look strong only after temporal shuffling.