A quantitative investment firm's ML team builds a three-factor return prediction model using features derived from price momentum, earnings revision signals, and NLP-scored news sentiment. The model achieves an annualized information ratio of 0.85 in backtesting. After eight weeks of live trading, the team discovers that the sentiment feature in the production serving pipeline uses a 1-hour rolling lookback window to compute a moving-average sentiment score, while the offline training pipeline used a 24-hour rolling lookback window for the same feature. The realized information ratio drops to 0.22 in live trading. This is the canonical example of which ML infrastructure failure?
Select an answer to reveal the explanation.
Short Explanation and Infographic
The model learned what a 24-hour average sentiment score looks like. In production, it's being fed 1-hour average sentiment scores. Those are different numbers even when the underlying news is identical. It's like training a weather model on daily temperature averages and then feeding it hourly readings — the model isn't wrong, the data pipeline is. A shared feature store with a single, version-controlled feature definition cures this by making it physically impossible for training and serving to compute the same feature differently.
Full explanation below image
Full Explanation
Training-serving skew is the condition in which the feature values seen during model training differ systematically from the feature values presented to the model during inference, due to differences in how the feature computation is implemented in the two environments rather than changes in the underlying data distribution.
In this scenario, the sentiment feature is defined as a rolling-window average of NLP sentiment scores over a news stream. The offline training environment (batch Python pipeline) was configured with a 24-hour window. The online serving environment (real-time streaming system) was separately implemented with a 1-hour window — either due to latency constraints, a miscommunication between the research and engineering teams, or a parameter that was never standardized. When the model ingests a 1-hour window score, it interprets the value using the statistical context it learned from 24-hour window scores (distribution mean, variance, sensitivity to equity returns), producing systematically miscalibrated predictions.
This is NOT concept drift: concept drift requires that the true relationship between features and labels changes over time in the world. Here, the relationship is stable — the problem is that the feature itself is being computed differently. It is NOT overfitting: an overfit model performs well on training data because it memorized noise; this model performs poorly because its inputs are structurally wrong. It is NOT look-ahead bias: look-ahead bias would mean the 24-hour window extended forward in time past the prediction point; a trailing 24-hour lookback is fully backward-looking.
The standard solution is a feature store — a centralized data infrastructure layer that stores a single, version-controlled implementation of each feature and serves it identically to both the offline training pipeline (via historical point-in-time snapshots) and the online serving pipeline (via real-time computation). Feast, Tecton, Hopsworks, and proprietary feature store implementations all enforce this principle: one feature definition, one computation, two consumers (training and serving). Any change to a feature's computation logic requires a version increment, a new training run, and an explicit deployment promotion.
In quantitative investment management, feature stores also solve the point-in-time correctness problem for financial data — ensuring that features computed for a historical training sample reflect only information that was available at that historical timestamp, preventing subtle forms of future information leakage.